<<< Updating the Spider Database
What is a Doorway Page? >>>
This chapter guides you through the process of creation of the robots.txt file.
Let's suppose you don't want German and French language spiders to visit your forum. All forum resources you want to protect from these spiders are located in the folder „/user/myhomesite/tools/forum", where „/user/myhomesite" is a root folder of your web site. You have to perform the following operations:
Start the program. If you are using it for the first time, a blank project will be created automatically. Otherwise the program will load the last opened project. In this case you will have to create a new blank project manually. To do this, go to the Project menu and select New.
Selecting robots
When a new project is created, you need to select German and French language spiders, which you want to keep away from your forum. You will select them at the Spiders List tab. The sequence of operations is as follows:
1. Select the item Select By... in the main Spider menu or in popup menu that shows up on the spider list. A window that allows you to select spiders by certain criteria will appear.
2. There is an option of selection by language in this window. Click on a button  to open the list of languages.
3. Select German and French on the list, and click on the Language field to hide the language list.
4. In the Selection Options filed, select Only these spiders. After these four steps, the window will look like this:
5. Press Ok to confirm your choice and all spiders matching these criteria will be selected automatically.
After you have selected the spiders, you need to specify the set of restrictions for them (i.e., what files and folders they are allowed to visit, if any). To do this, switch to the tab marked Disallow.
The left part of the tab contains all selected spiders. There is also one fake spider that encompasses all selected spiders. On the list, it is called All Spiders. You may regroup selected spiders by country or language, or just leave them sorted as they are. The sorting mode may be selected from the combobox above the spider list.
The tree of folders and files of your web site is displayed at the right part of the tab. To view the tree of your site, you'll need to specify its location by pressing Select Location... button or selecting Site Location item from the Disallow menu. A window asking you to specify the location of your site will show up.
Select FTP Server and fill in the fields required to establish the connection: Host, Port, User Name, and Password. In the Folder field, enter „/user/myhomesite/". You may save these settings in the program by pressing Save as Default Settings button. If you are not sure about the exact path to your site folder, press FTP Browse button to browse the server and select your folder. After you fill in all the data, the root folder selection window will have the following appearance:
When all data is in place, press Ok to set your main folder.
Please mind that the root folder of your site is not always the same as root folder on FTP server. If you set the wrong folder, spiders may index the pages you wanted to protect.
1. In the Group by: field, select None. This means that selected spiders will not be divided into smaller groups (because in our case there's no difference between German and French language spiders).
2. Select all spiders except the fake one "* (All Spiders)". You can do this using Shift and Ctrl keys together with your left mouse button.
3. Then go to the folders and files tree at the right. Open the root folder (click on '+'). When all folders and files are loaded into the tree, open the tools folder. Tick the box next to the forum folder.
After you set the restrictions, press the Generate file button to create the robots.txt file.
After the registration check, a window with newly created robots.txt file will appear. To save this file, press Save file.
The file robots.txt will contain such strings:
# Generated by Robot (http://www.dealer.com)
User-Agent: *
Disallow:
User-Agent: AbachoBOT
Disallow: /tools/forum/
User-Agent: Acoon
Disallow: /tools/forum/
User-Agent: antibot
Disallow: /tools/forum/
User-Agent: motor
Disallow: /tools/forum/
User-Agent: DeepIndex
Disallow: /tools/forum/
User-Agent: blindekuh
Disallow: /tools/forum/
User-Agent: Firefly
Disallow: /tools/forum/
User-Agent: francoroute
Disallow: /tools/forum/
User-Agent: Marvin
Disallow: /tools/forum/
User-Agent: xyro
Disallow: /tools/forum/
User-Agent: Tarantula
Disallow: /tools/forum/
User-Agent: newscan-online
Disallow: /tools/forum/
User-Agent: Pompos
Disallow: /tools/forum/
User-Agent: search.ch
Disallow: /tools/forum/
User-Agent: speedfind
Disallow: /tools/forum/
User-Agent: suchbot
Disallow: /tools/forum/
User-Agent: Suchknecht.at-Robot
Disallow: /tools/forum/
User-Agent: CoolBot
Disallow: /tools/forum/
User-Agent: SynoBot
Disallow: /tools/forum/
User-Agent: Tigersuche
Disallow: /tools/forum/
User-Agent: verticrawl
Disallow: /tools/forum/
User-Agent: VoilaBot
Disallow: /tools/forum/
User-Agent: SurferF3
Disallow: /tools/forum/
User-Agent: marvin/infoseek
Disallow: /tools/forum/
User-Agent: Webspinne
Disallow: /tools/forum/
User-Agent: WOLP
Disallow: /tools/forum/
Save the generated file by pressing Save file button. A standard 'save file' window will pop up, where you'll need to specify the file name and the folder to save the file to. If you want to upload this file to the server, you may only save in as "robots.txt". You may also want to create a separate folder to store your robots.txt files.
If you are using demo version, you will not be able to save the generated file.
If you try to follow this example, some strings in your robots.txt file may be different from those shown above. This may occur due to the fact that spider lists are updated regularly.
The program allows you to upload the newly generated file to your server easily. To do this, use FTP Uploader. Switch to this module and enter your connection data: FTP Host, FTP Port, FTP Username, and FTP Password. You may just import the previously saved settings by pressing Load Default Settings button.
Press Connect to establish a connection with your FTP server. Then switch to the home folder of your site and copy the new robots.txt file there.
When uploading is finished, press Disconnect.
<<< Updating the Spider Database
What is a Doorway Page? >>>
|
