Skip to content

Work with robot.txt

Your site content is very important for your Site . The site crawl take most important part of your site , robot.txt tells search engine how they should visit your site .If you want to block specific search engine & also directories. So by the help of robot.txt you can do the job easily .The robot.txt file are text file which are placed on your web-server . They must be placed on the root folder, as an example…


If your website does not have a robot.txt file then this is what happens –
A robot comes to visit. It looks for the robot.txt file. It does not find it because it isn’t there. The robot then feels free to visit all your web pages and content because this is what it is programmed to do in this situation.

Create robot.txt like that 

User-agent: *

In disallow you place that folder or search engine name which you want to block. SEE THIS   And User-agent: * means apply on all robots


  • Didn’t block any popular search engine for Ex. Google, MSN etc.
  • Use allow tag for your site .

Previous article

Generate Your Visualized Data

Next article

WordPress Child theme

Join the discussion

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: