<<< Spiders List Tab
Log Analyzer Tab >>>

The Disallow Tab consists of:

• a list of selected spiders,
• a combobox where you can specify the grouping criteria for these spiders,
• a tree of folders and files on a local PC or a remote FTP server, starting from a folder specified in the Home Page Settings window,
• a checkbox that will completely disallow access to the site for selected spiders,
• a button that calls up the Home Page Settings window,
• a button that resets all restrictions,
• and a button that calls up a generator window for the standard instructions file.

You may perform the following operations for spiders you have selected in the previous window:

• disallow access to certain files or folders,
• allow access to the entire site,
• allow access only to certain files or folders.

The general folder tree can be expanded by opening subfolders.

There is always an entry marked '* (All Spiders)' on the selected spiders list. It corresponds to the '*' directive in the standard instructions file that sets restrictions for all existing spiders. Selected spiders on the list are sorted alphabetically.

If you select a spider and right-click on it, you'll see a popup menu that contains a string Spider <Spider name> Homepage. If you click on it, the spider's home page will be opened in the system default browser.

Grouping spiders

Grouping is disabled in all new projects and spiders are listed alphabetically. You may group them by language or country. When you revisit the Disallow Tab, spiders are grouped according to the mode selected during the previous working session at this tab.

You may set the restrictions for every single spider on the list as well as for groups of spiders. To select all spiders by country or language, just select this country or language. All corresponding spiders will be selected automatically. For customized selection, use Shift and Ctrl keys.

In case when certain spiders in the selected group already have some restrictions, the restrictions will be shown as set for the group only if they apply to all spiders in this group. When you add or remove restrictions for a selected group, they will be added or removed for all spiders in this group.

The checkbox of total access restriction is always unchecked, but this does not mean that if a spider that had it checked is included in the group, it will be unchecked for this spider as well. If this checkbox is checked for the group, access to all pages of your site will be forbidden for all spiders in the group.

Setting the restrictions

When you start the program, create a new project or open an existing one, or when you change the Home Page Settings , the folder tree contains only one folder, which is the root folder of your site. To open the tree of subfolders and files of nesting level 1 in this folder, click on the plus sign next to the folder name or double-click on the name itself. If there is no tree for this folder yet, it will be built when you open it. If the contents of the site, for which the standard instructions file is being generated, are located at a remote FTP server , the tree building process will be reflected in corresponding modal window, where you may interrupt the process at any moment by pressing Cancel.

To close the folder, click on minus sign next to its name. When there is neither plus nor minus next to the folder name, this folder is empty.

Each subfolder and each file in the tree have checkboxes next to them. You may add any file or folder to the list of restricted areas by simply ticking the appropriate box.

Once a tree has been built successfully for a given folder, it would not be rebuilt every time you close and reopen this folder.

The program also allows you to set the restriction for a given file format. To do this, you need to select Disallow File Extensions item in the menu Disallow. A window will appear, where you'll be able to specify extensions of files you want to disallow access to. If you check the Disallow files with extensions box and press OK then a list of extensions will appear in each folder before files and subfolders. Each extension will also have a restriction checkbox next to it. To hide the extension list from the subfolder tree, select Disallow File Extensions again and uncheck the box Disallow files with extensions.

However, not all robots support restriction by extension. Therefore this method does not give you a 100% warranty that documents you have protected will not be indexed.

When you select an element from the tree and press Reload, the branch of the tree that contains this element will be rebuilt. You may use this button to refresh your tree whenever necessary.

Removing the restrictions

You may deselect all branches in the tree (i.e., remove all restrictions) for a given spider or group as well as for all spiders on the list. Just press the Uncheck... button and select the appropriate mode in the popup window: Selected spiders or All spiders on the list.

When you press Generate File button, a generator window for the standard instructions file will show up.

Setting site location

This window allows you to specify the parameters of the main page of web site, for which a standard instruction file is created.

You may select either local PC or a remote FTP server, where the site content is stored.

When working locally, you need to specify the path to the website root folder. Just press the Browse... button and select the appropriate folder in the Folder selection window, or type in the path in the space provided (see screenshot):

This window allows you to specify the local folder in which the site content is stored. A standard instruction file is created for this folder. The window contains a combobox for selection of local disk, a window that displays folders tree on the selected disk, and a field that contains path to the selected folder. When you press OK this path will be recorded into the Folder field of the Home Page Settings window.

When working with a remote FTP server, you need to enter the address of your FTP server, e.g., ftp.server.com, port number (default 21), user name and password for access to the FTP server as well as path to the root folder on the server, where the site contents are stored. The path should start with slash (/), e.g., /pub/MySite/. If you are not sure about the exact path to the main page, press FTP Browse button. A window will show up, where you'll be able to select the required folder. When the required folder is selected, press OK button, and the Folder field will be filled in automatically.

The default values of User Name and Password fields are set to anonymous.

When creating a new project, default settings are 'local PC' and program installation folder.

When working with a remote FTP server, you can save the connection settings as Default values and load them in other areas of the program as needed. This can be done easily using the Save as Default settings and Load Default settings buttons.

Generating robots.txt file

Once you have configured all settings, you can press Generate robots.txt to generate your exclusion file. A pop-up window will appear allowing you to see the newly created robots.txt file exactly as it will be uploaded to your FTP server. You may also modify this file and save it onto the disk.

You may save the newly generated file using the Save Filebutton.

Two first lines in the generated file contain the program name, developer's site URL, and URL of the site for which the file is being created. These lines start with # and they are not examined by web robots and spiders. Web robots and spiders view all other lines that do not start with #. Based on this information, spiders will have the directives for conduct at your site.

You may edit the file after creation, right in the window that displays a newly generated file: add comments, optimize the text, etc.

If you are using an unregistered version, you will not be able to save the newly generated file on the disk, as well as edit/optimize the generated text.

Disallow file extensions

You may specify extensions of files you want to exclude from being indexed by spiders. If an extension you need is not on the list, you may add it. To do this, enter the extension in the Extension field, without an asterisk (*) and dot (.) before it and click on the Add button, or press Enter . When you don't need a certain extension on the list, you may get rid of it just as easily. Simply select this extension and press Delete. Should you want to clear the whole list, press Delete All. The extensions from the list will be displayed in the folder tree at the Disallow tab, if you check the box Disallow files with the extensions.

If you do not want to set any restrictions according to the filename extension, uncheck the box Disallow files with the extensions.

To confirm the changes, press Ok; to reject the action, press Cancel.

<<< Spiders List Tab
Log Analyzer Tab >>>