Robots.txt - Help - doorway page management

Some of you may ask: "How do I prevent Search engine #1 from indexing the page designed for Search engine #2?" The answer is simple: use the 'robots.txt' file. There are also other reasons to disallow indexing of certain or all pages on your website by certain search robots.

Supposing that you create different versions of the same doorway page or another page and each search robot indexes each copy of this page, in theory, you may be accused of spamming. It is widely known, for example, that AltaVista especially hates all kinds of duplicates or pages with similar content. Therefore, if you create many pages that are very similar to each other, you risk receiving a red card from leading search engines. As a matter of fact, most people do not really need to worry about many similar pages indexed by the same search robot, because these people do not create so many duplicated pages. If your pages vary substantially in terms of volume and content, you also have nothing to worry about.

You will also avoidall potential problems if you concentrate primarily on fine tuningthe pages that already exist at your site and have unique content, instead of creating scores of new pages with similar content. Some people simply submit specific pages only to the engines for which they had been optimized. This may be the easiest method to avoid accusations of spamming search engines. This one may work. But any other robot may find this page, even though it wasn't submitted for registration at that engine.

If you want to create many doorway pages, which will inevitably have similar content, and optimize them for different search systems, you must use the robots.txt file.

Let's suppose you have created three groups of doorway pages for three different search engines: Google, Yahoo, and Inktomi, and you placed these pages in three different folders: "/doorway/google", "/doorway/yahoo", and "/doorway/inktomi". Now you need to arrange things in such a way that each search engine could only see 'its' pages. Let's take a look at the process in details:

Step 1.
Start the program and create a new project.
At the Spider List tab, select the following spiders: "Google", "Yahoo", and "Inktomi", then switch to the tab Disallow.

Step 2.At the Disallow tab, you need to specify the root folder of your site. Press the Select Location button or click on Site Location item in the menu Disallow. Then press Ok

Step 3.
At the left part of the Disallow module there is a list of spiders. In our example the list consists of four elements: "* (All Spiders)"; "Google (Googlebot)"; "Inktomi (Slurp)"; "Yahoo (YahooBot)".
Select Google and disallow access to folders "/doorway/yahoo" and "/doorway/inktomi". To disallow access to a folder, simply check the box next to its name.
Then select Yahoo and disallow access to folders "/doorway/google" and "/doorway/inktomi".
And naturally, disallowaccess to "/doorway/google" and "/doorway/yahoo" for Inktomi.
Since there are many more spiders beside these three, we recommend that you disallow access to two of three folders to the spider "* (All Spiders)" as well.

Step 4.
Now you need to generate the robots.txt file. To do this, simply press Generate file. The file you created will be displayed in the popup window. You may make some changes in this window, if necessary. Then press the Save file button to save the newly generated file.

The Save function is not available in the demo version.

The program will generate the file like this:

# Generated by Robot (http://www.dealer.com)

User-Agent: *
Disallow: /doorway/inktomi/
Disallow: /doorway/yahoo/

User-Agent: Slurp
Disallow: /doorway/google/
Disallow: /doorway/yahoo/

User-Agent: YahooBot
Disallow: /doorway/google/
Disallow: /doorway/inktomi/

Only the /doorway/google resource remained open for all spiders. Since Google had the same restriction set as the fake spider ('*'), the restrictions for Google were not specially included into the file.

Step 5.To upload the newly generated file to your server, use FTP Uploader. Switch to this module and enter your connection data: FTP Host, FTP Port, FTP Username, and FTP Password. You may simply import the previously saved settings by pressing Load Default Settings.
Press Connect to establish a connection with your FTP server. Then switch to the home folder of your site on the server (e.g., /user/myhomesite/) and copy the new robots.txt file there.
When uploading is finished, press Disconnect.

The file you upload must be named robots.txt, all letters in lowercase. If you do not observe this requirement, spiders may be able to index all your pages freely.

The "robots.txt" file must be located in the root folder of your website. In any other area of the site this file will be simply ignored. If you do not observe this requirement, spiders may be able to index all your pages freely.

<<< What is a Doorway Page?

support@net-promoter.com