Here I have picked one of most interesting topic for API development. From many year, we are developing an API in two Protocols REST and SOAP and most of time developers are preferred REST, because of its simplicity and easy to handle in APPS'. But last many days I have seen a new Query language with more simple and fast as compared to REST and it is graphical query lang.(GraphQL).Intro...
Your site content is very important for your Site . The site crawl take most important part of your site , robot.txt tells search engine how they should visit your site .If you want to block specific search engine & also directories. So by the help of robot.txt you can do the job easily .The robot.txt file are text file which are placed on your web-server . They must be placed on the root folder, as an example… www.yourwebsite.com/robots.txt
If your website does not have a robot.txt file then this is what happens –
A robot comes to visit. It looks for the robot.txt file. It does not find it because it isn’t there. The robot then feels free to visit all your web pages and content because this is what it is programmed to do in this situation.
Create robot.txt like that
User-agent: * Disallow:
In disallow you place that folder or search engine name which you want to block. SEE THIS And User-agent: * means apply on all robots
- Didn’t block any popular search engine for Ex. Google, MSN etc.
- Use allow tag for your site .