Robots.txt - Help - log profile manager

Running a professional website usually means serious expenses. When effectiveness of different business activities (like sales, advertising, customer relations, etc.) depend upon the Web, decision makers must possess comprehensive data on results of website operation. A server can record numerous parameters of each client request session into so-called log file.

Formats of log files

The structure of log file may differ greatly from case to case. First of all, it depends on the type of server you are using. At the present moment, the most popular severs are Apache 2.х, Microsoft IIS, and Zeus.

There are several log formats in which the data on spider visits may be recorded. These are Combined Log Format (used by Apache web server), W3C Extended Log Format (Microsoft IIS), and Definable Log Format (Zeus).

Here is an example of the logfile string in Combined Log Format:
164.68.87.21 - - [37/Feb/2003:07:54:28 -0600] "GET /register/index.pl?action=buy HTTP/1.0" 200 12324 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" "www.mysite.com".
We can see that the spider of Google search system has visited the site. You can review the detailed report on the Apache web server and logfile formats at the Apache official site.

The W3C Extended Log File Format differs greatly from other formats. The file in this format consists of a set of strings built of ASCII symbols, each string ending with LF or CRLF. The information on visits is preceded by strings starting with #, which contain directives. The Fields directive is the most important; It specifies the parameters recorded into the log file. Directives are followed by strings composed of fields that correspond to one HTTP transaction. Fields are separated with spaces or tabs. If a field is blank, it may be replaced with the minus symbol „-„. You may review the complete list of field specificators used in the log file, at the W3C official website. Below is a fragment of a log file in this format:

#Software: Microsoft Internet Information Services 5.1
#Version: 1.0
#Date: 2003-04-07 08:41:55
#Fields: date cs-method cs-uri-stem sc-status cs(User-Agent)
2003-04-07
2003-04-07 GET /iishelp/iis/htm/core/iicodirc.htm 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1)
2003-04-07 GET /iishelp/iis/htm/core/iiverid.htm 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1)

The Zeus web server may record log files in two formats: User Definable Logging Format, and Binary Log Format. The User Definable Logging Format is extremely flexible, it is entirely configured by the user with the aid of so-called Logging Macros. The list of all applicable macros is published at the Zeus website. Default macros for recording log files are:
%T|%h|%M %U|%{Content-Type}o|%s|%u|%b|%{Cookie}i|%{Referer}i|%{User-Agent}i.
A string that corresponds to this log file -
970901-142349|machine.zeus.co.uk|GET /top.gif|image/gif|200|-|11940||http://www:1234/|Mozilla/5.00 [en] (WinNT; I).

Creating a new profile for Combined Log Format

1. Selection of fields ordering in the log file. The list of fields is displayed in the left part of the window. You may arrange them by selecting fields and clicking on "up" and „down" arrows, or dragging them with your mouse. The program selects fields that are used in the log file. Certain fields are obligatory for proper operation of the program and you can not de-select them. To make your task easier, we have provided a possibility to preview the results in visual form in the Template of Profile field. In this field, all fields from the log file are displayed in square brackets ("Spider IP" would be shown as [Spider IP]).

2. Additional symbols in the log file. Additional symbols are used in the case when your log file format uses field delimiters. For example, the Spider IP field may be delimited by "##" symbols. In such case you enter these symbols and tick the Before and After boxes, select Spider IP from the fields list and press the Insert button. Again, you may preview the results in Template of Profile field.

3. Saving the profile. To save your log profile, press the Save button. A window will appear asking you to enter the name for this profile. You can not save your profile as "Apache 2.0 standard" or "Zeus Server v1 standard", because these profiles are read-only.

Creating a new profile for Definable Log Format

In the macros string, enter the sequence of macros specified in your web server settings. Instead of typing macros in manually, you may select the ones you need from the list and press Add Macros button. Certain macros are obligatory. You will not be able to save or test your profile until all obligatory macros are in place - the program will hint you, which one is missing.

Testing your profile

You may test the profile you have created. This feature is only available when editing a profile. To start testing, press the Verify button. In this window, you may type in a string of your log file manually or open an existing log file and pick the string you want from it. The program will check it and inform you, whether the string matches your profile.

Editing an existing profile

You may edit previously created profiles (except for Apache 2.0 standard and Zeus Server v1 standard, which are read-only). To start editing, select the profile from the list and press Edit , or double-click on the profile name.
The recommended operation sequence is the same as when creating a new profile.

Cloning an existing profile

Select a profile from the list of available profiles and press Clone. Then enter a new name for the copy of this profile. The copies of read-only profiles may be edited after cloning.

Deleting a profile

Select a profile from the list of available profiles and press Delete or hit the Del key You can not delete read-only profiles.

<<< Report Wizard
Configuring User-Agent List >>>

support@net-promoter.com