<<< Welcome to Robots.txt Editor
Features >>>
Beside the conventional visitors who come to your site to buy something or simply search for the information, your site is also visited by robots - special search programs that crawl the Internet to index web pages and add them to search engines' databases. The robot's activity gives your site greater visibility and raises traffic. However, your site might contain:
- private information
- pages designed only for clients or staff
- pages in different languages that are not supported by some search engines
- web pages optimized for specific search engines
- multimedia files and e-mail addresses
Naturally, you may want to restrict the access to these resources. You also shouldn't forget that:
- excessive spidering overloads your server;
- indexing of the whole site may jeopardize your ranking on search engines;
- certain spiders index the first two or three pages only.
The Robots Exclusion Protocol is a method that allows Web site administrators to indicate to visiting robots which parts of their site should not be visited by the robot. When a Robot visits a web site, it first checks for the file robots.txt in the root directory; e.g. http://mysite.com/robots.txt. If it can find this file, it will analyze its contents to see if it may retrieve further documents (files). You can use this program to customize the robots.txt file to apply only to specific robots, and to disallow access to specific directories, files, or even file types.
<<< Welcome to Robots.txt Editor
Features >>>
|
