Even there are some great services out there like statcounter and google analytics for providing real time visitor information for your site but its still usefull to analyse the information in your webservers logs in addition to these.

As the visitors to your website increase so does the size of your log files. It has been estimated that they grow by 1MB for every 10,000 hits. This may not appear to be much of a problem but be aware that a 'hit' here means a log entry for every item that apache serves eg. images, css files, pdfs etc so just loading one page can have multiple log entries.

Ideally you want to reduce your current logs to managable files for example; one file for each month. When apache is running it opens the log file and holds it open for writing. Reducing the file sizes can have performance benefits and also make it easier to parse the log files for analysis.

1. Tell apache what to log

The first step is to setup the format of our log files in httpd.conf, we will use the combined log format as this is prefered by most log file analyzers and indeed recommended by apache. The LogFormat directive allows us to specify the format of the log file entries and to give that format a name, in this case 'combined'.



As mentioned earlier, apache will log every media item that it serves. It would greatly decrease the size of our logfiles if we could tell it to ignore items such as images. This is achieved by using the SetEnvIf Directive. The SetEnvIf directive allows us to set an enviroment variable on the basis of a request that we can use as a filter when configuring the actual log file for our host. The basic format for SetEnvIF is as follows



The Attribute part can be one off the following:



Match Condition is a perl compatible regular expression and the last section allows us to set the value of our enviroment variable which can take 1 of three actions



Now we tell apache not to log any images or other files that we don't want to count as pages.



Essentially we are asking apache to check the incoming URI and if it marches any of the above extensions to set an enviromental variable called 'dontlog'. We don't need to assign a litteral value to this variable as later when we decide to log or not we only need to check the existance of this variable.

Please note the use of SetEnvIfNoCase instead of SetEnvIF. This works in an identical fashion to SetEnvIf except it is case insensitive which is good for us as people may name their images .JPG. This also removes the pesky request for those damn favicons from appearing in your logs!

2.Log Rotation

In its default configuration apache will log all requestes to one log file which will keep growing over time. As mentioned before we really want to reduce this to be more managable. Essentially what we want to do is on a specified date/time, stop logging to the current log file, create a new logfile, backup the old log file and continue logging to the new one. The only  major stubling block to this is the fact that apache is holding the log file open for writing, so to perform any opperations on that file we must stop apache first, not great if you are running 24-7!

Apache provides us with the use of piped logs to help us get round this. Piped logs give us the ability to pass the output of our CustomLog directive to another program using the pipe opperator '|' which by-passed the need to open the log files as the rotate logs program recieves the request from apache and deals with the writing to file.

Consider the following CustomLog directive;



This is writing to the specified logfile (/path/to/my/logfile/access_log) in the combined format that we specied earlier and ignoring anything request that has had the dontlog env variable setup that we configured earlier.

We now need to pipe this to our log rotation program. Apache comes with its own program called logrotate but i will be using cronolog as it allows better seperation (by date) of log files.

cronolog is available here

Once you have downloaded the latest version unpack the archive:



switch to the newly created directory and issue the following commands:



then switch to root user and run



once you have installed cronolog (default location is /usr/local/sbin) we need to change our custom log directive



This is now piping the log output to cronolog. I have specified cronolog to rotate my logs once a month by using "%m-%Y" this will place the new log in a directory created in the following format MM-YYYY eg:



you can specify a host of other date/time combinations which are shown on the cronolog website

once you have made these changes to your httpd.conf remember to restart apache so they take affect

3. AwStats

Awstats is a logfile analyzer. I also played around with webalizer which does a similar thing but in a toss-up i chose awStats.

awStats is available here.

once you have downloaded and unpacked awStats you must move the contents of the unpacked directory to /usr/local/awstats



then go to /usr/local/awstats/tools and run the configuration script



this will create a configuration file in /etc/awstats if you picked all the default options in the config script. If you entered www.mydomain.com as the website name  the config file will actually be called /etc/awstats/awstats.www.mydomain.com.conf, however throughout the rest of the setup it is generally refered to as www.mydomain.com (awstats will add the awstats. and the .conf around it) NB. Please substitute www.mydomain.com with your real web address!

open up this config file and check the following lines



where LogFile is the location of our logfile, "%MM-0-%YYYY-0" is telling awstats to read the logfile in the right directory that we specified earlier. the -0 is the time where 0 = now, thus look in directory currentMonth(MM)-currentYear(YYYY). LogType is W for web (as opposed to FTP, Squid etc). LogFormat is 1 for the combined log format we setup earlier. SiteDomain is the domain for your site. DataDir is the location where awstats will store its datafiles, this directory needs to be writable by the webserver and defaults to /var/lib/awstats.

Once you have configured the .conf file its time to parse the logfiles and build the output, use the following command



if you have several sites then copy/paste a new .conf file in /etc/awstats changing the above sections for each domain, multiple configs can be run with one command



when this is done we should be able to see the output for each domain  by browsing to:



where config=mydomain.com is the name of the config file we created earlier, this can be changed depending on which of your configs you wish to view (if you have more than one domain).

Now we need to add a cron entry to update this daily.



This will update the stats every day at 4am

if you don't want to use the web browser to access awstats.pl directly (as this may increase load on the server) you can generate static reports using the following command



you will then have to add another cron entry to build these at regular intervals.

HTH!