back | forward

Spider your site to gather information about your web sites structure, pages and links to outside pages. Why do this? Firstly, different pages of your site may contain different types of information. When submitting your web site to search engines you will probably want this information to be indexed and catalogued in the search engines databases. Secondly, pages of your web site contain links to outside sources, and the latter also need to be indexed - this may diversify and increase your page's visibility on search engines.

If you want to spider your web site you will need to provide the following information about it.

• Start URL. Usually it is the site's main page, as it contains links, following which a visitor may get to any part of the website. Starting from this page spider will index the rest of the site.
• You may want to index external links on your web site. For this option select Use links to external pages
• Define file extensions to index. You may select certain file types to be indexed by checking appropriate check boxes in the Extensions window.
• Set Maximum number of links and maximum number of levels to be spidered. It is recommended that your site should not contain files that are not within two clicks of the mouse, i. e. two levels.
• In the URLs window you will see page URLs, found during spidering. You can select delete all URL or several URLs by pressing Delete All or Delete selected respectively. To select multiple URL click on them holding Ctrl.

To start spidering automatically, enter the URL of your opening page into the Start URL field and click Add and Spider URL. After getting the list of URLs you may go to the Submission tab - click Submit.

back | forward