<<< Using robots.txt Files for Doorway Page Management
Importing Files >>>
Website owners often face a lot of questions concerning indexing, like: how often search engine spiders visited my website, what pages they indexed and when exactly that happened, etc. Answers to these questions may be found by analyzing web server working log files.
Importing log files is a first step in the process of analysis. After the files are imported, the program builds a tree displaying spider visits.
If you want to see what pages have been indexed, you need to select Path-Spider-Date representation from the list at the right. There are 6 representation forms: these are all possible combinations of 3 fields: Spider, Date, and Path. To rebuild the tree after you change the representation form, press Rebuild Tree.
When you select any third-level node (terminal node) in the tree, you can see the spider's request for the selected node at the bottom of the window. When you select a second-level node, a list of requests will be displayed at the bottom. The information in key fields of these requests will match the selected node and sub-nodes.
To make visual evaluation easier, pages indexed by spiders are displayed as different icons in the tree, according to the request and its completion status. There are different icons for:
 successful requests to robots.txt;
 requests to robots.txt that resulted in errors;
 requests to graphic file;
 requests that were completed successfully;
 requests that resulted in errors;
 requests that resulted in redirection;
Spiders are also displayed by different icons
 - those that addressed robots.txt (during the day);
 - those that did not address robots.txt (during the day);
Tree cross icon with red frame contains spider visits to disallowed pages.
After some time spiders return to the same pages several times. If you would only like to see the last visit, select Show only the last visit, and press Rebuild Tree .
If you need more details on a given spider request, select the node that corresponds to this request in the visits tree. The required information will be displayed in the text box below. The text box has context menu with items: Copy and Copy All, that allow you to copy the selected fragment or all contents of the box, respectively.
If you need a complete information on a given spider that indexed your page, right-click on the corresponding node and select Spider Info.
You may need the generalized information on indexing of your site, rather than specific spider visits. In this case you would want to create a report. You will find instructions for creating reports here.
The program also offers a handy feature of exporting information on spider visits into other format of your choice. See an example of exporting a log database into xml format.
<<< Using robots.txt Files for Doorway Page Management
Importing Files >>>
|
