How to analyze the access logs of a web server with AWStats

AWStats is a log analyzer application written in Perl, that summarizes valuable information about the traffic received by a web server, and displays it in a web frontend with data in tables and graphs.

This post explains how to install and configure AWStats in an apache web server running on a linux (Debian)  machine. The official documentation of AWStats is available at http://awstats.sourceforge.net/docs/

AWStats versus Google Analytics

AWStats generates statistics about visitors, visits, pageviews, and detailed reports by type of browser, country, etc… much like Google Analytics does. The difference resides in the fact that AWStats does not require adding javascript to the pages, and therefore is able to anlyze all kind of traffic, even traffic generated from browsers with no javascript support (including many crawlers that can generate a significative load on our server).

AWStats installation

We start by downloading from  the AWStats home page at sourceforge the ‘awstats-7.0.zip’ package.

We must uncompress the zip file under ‘/usr/local’, and rename the resulting ‘awstats-7.0’ directory as ‘awstats’. Next, we run the ‘awstats_configure.pl’ configuration script under the ‘awstats/tools’ subdirectory.

The complete log of the initial configuration is as follows:

1. The script requests the location of the web server configuration file:

2. The apache configuration file is modified by adding the required directives to give access to the AWStats frontend:

3. A configuration file for our main site is created.

We will see later that this script needs to be edited to match the details of our web server setup.

4. The script informs us that the update process should be automated by means of a cron entry.

Once the initial configuration is done, we must create the directory where the AWStats database will be located:

With this, we can now point our browser to  ‘http://localhost/awstats/awstats.pl?config=www.openalfa.com’ , and give a first look to the AWStats frontend:

 

Of course, all data are zero in this first page, because we have not processed yet any log file. But we can already browse the variety of reports that AWStats will make available with a single click from this page.

 

Processing of log files

Processing of a log file is performed by running the script  “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.openalfa.com”. But first, we need to edit the configuration file for our site, to specify the location of the log files and how they are named.

Our server creates daily log files with filenames in the format ‘access_log.YYYY-MM-DD-00_00_00’, under ‘/var/log/apache2/openalfa’. For instance:

Therefore, we must edit the ‘/etc/awstats/awstats.www.openalfa.com.conf’ configuration file, and replace the line ‘LogFile=”/var/log/httpd/mylog.log”‘ with:

With this setting we ensure that, every time the update script is run, the log file to be processed will be that of the day before the day when the update script is run.

Once the configuration file has been edited, the update is performed issuing the command:

going back to the browser and reloading the awstats home page, we can confirm that the first processed data are present in it:

 

Finally, we just have to automate the update process to be run on a daily basis. We do this by adding a cron entry like this:

 

Processing of old log files

Most likely, by the time we perform the AWStats installation there is aready a set of old log files from the days, months and even years since the web server started. If we want to process these historical data, we must follow these steps:

1. Delete the AWStats database just created in the first run of the update:

2. Edit the configuration file ‘/etc/awstats/awstats.www.openalfa.com.conf’ to specify the log data to be processed as:

The update process will read the output from the ‘logresolvemerge.pl’ script included in the AWStats package.

This script reads all the log files given as argument (in the example above, all ‘access_*’ files under the  ‘/var/log/apache2/openalfa’ directory), and will send them to the update process, sorted by the dates in the log entries in them.

3. Execute the update process

4. Edit again the configuration file to revert the value of the LogFile setting:

 

How to configure AWStats for several sites in the same web server

Our web server might be giving service to more than one site (for instance, by using ‘VirtualHost’ directives). In our case the three domains www.openalfa.com, blog.openalfa.com and blog-en.openalfa.com are served by the same web server.

The configuration of AWStats in this case is performed by creating under the ‘/etc/awstats’ directory a configuration file for each domain. Inside the configuration files,  the ‘LogFile’ setting will specify the location and format of the log files for each domain.

Then, instead of creating a separate cron entry for each domain, we create a single entry to process all of them like this:

For each of these domains, if we want to process old log files previously existing, the same considerations exposed in the previous section apply.

—