Robots.txt Editor Overview >>>

Before indexing your site, the spider downloads robots.txt file that contains instructions on what should and what should not be indexed. Therefore a key to controlling spiders is in creation of that file. If you have a large website of update it frequently, creation and editing of this file will be a hard and dull work.

Robots.txt Editor is an easy-to-navigate visual editor that will enable you to specify different directives for selected spiders in specific areas of the site and generate the robots.txt file quickly and easily. However, you should also be aware of the fact that robots do not necessarily have to obey the indexing instructions set forth in robots.txt. Therefore the file itself will not be of great use as long as there is no possibility to check the real actions of a given spider for a specific web page. All spider visits are recorded in a log file stored at your server. Again, you may review all log files manually and compose your database. But why should you dig yourself into this dull and time-consuming task? You may take advantage of our Spider Robots.txt module that will allow you to download and save log files with information on spider visits, as well as export it into different file formats. Besides, this information will be structured. You'll be able to track the visits by spider, by time period or by specific pages, and to generate a report by various criteria.

Robots.txt Editor Overview >>>