Install Webalizer (basic, working setup):

Before beginning the install, it should be noted that it will be helpful to have a consistent directory structure for all users/domain hosting customers. While nothing is set in stone, pick a directory structure and stick to it consistently. A common structure is to put virtual domains in "/usr/home/." For example, "virtual domain 1" would be put in /usr/home/vdomain1, and its web accessible html files would be stored in "/usr/home/vdomain1/www/data" (the data directory is also called "htdocs" by many hosting companies). A visual representation would be:

To continue with our above example, Apache log files could be stored in /usr/home/vdomain1/log, and Webalizer data could be stored in a web accessible directory, for example, /usr/home/vdomain1/www/data/stats.

Installing Webalizer and configuring Webalizer is relatively easy. The main points that need to be considered are getting the latest port of Webalizer. If the port that came on the FreeBSD 6.0 install iso is used, it will exit out with the following error:

=> Attempting to fetch from http://manager.mt-plugins.org/.
fetch: http://manager.mt-plugins.org/PluginManager-0.1.8.tar.gz: No address reco rd
=> Attempting to fetch from http://mt-plugins.org/.
fetch: http://mt-plugins.org/PluginManager-0.1.8.tar.gz: Moved Temporarily
=> Attempting to fetch from ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/.
fetch: ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/PluginManager-0.1.8.tar .gz: File unavailable (e.g., file not found, no access)
=> Couldn't fetch it - please try to retrieve this
=> port manually into /usr/ports/distfiles/ and try again.
*** Error code 1

Stop in /usr/ports/www/MT-PM.
*** Error code 1

Stop in /usr/ports/www.

To update your ports files, go here. If you aren't using CVSup to upgrade your ports, you can go to the FreeBSD.org ports collection and manually mirror all of the files into your /usr/ports/www/webalizer directory. A Makefile with a revision number of 1.52 or higher should be used.

The other important point of the installation is, as already mentioned, to come up with a standard directory structure for both Webalizer and your users' Apache log files.

After the updated files are in place, to install Webalizer:

Issue commands:

Make
Make Install

The install takes only five to ten minutes.

If you intend to use Webalizer in an Apache virtual host situation, you will need to configure your virtual hosts so that a log file is generated for each virtual host. The basic setup for this is to add tags to the virtual host section of your Apache httpd.conf file for each virtual host on the server:

ErrorLog "/usr/home/vdomain/log/error.log"
TransferLog "/usr/home/vdomain/log/access.log"

For example, if there were three virtual hosts on the server, the httpd.conf file should include something similar to the following, and containing the tags above, which will generate the virtual host log files in the user's home directory:

<VirtualHost 65.201.93.140:80>
DocumentRoot /usr/home/vdomain1/www/data
ServerName www.vdomain1.com
ServerAlias vdomain1.com *.vdomain1.com
ErrorLog "/usr/home/vdomain1/log/error.log"
TransferLog "/usr/home/vdomain1/log/access.log"
<Directory "/usr/home/vdomain1/www/data">
allow from all
Options +Indexes
</Directory>
</VirtualHost>

<VirtualHost 65.201.93.140:80>
DocumentRoot /usr/home/vdomain2/www/data
ServerName www.vdomain2.us
ServerAlias vdomain2.us *.vdomain2.us
ErrorLog "/usr/home/vdomain2/log/error.log"
TransferLog "/usr/home/vdomain2/log/access.log"
<Directory "/usr/home/vdomain2/www/data">
allow from all
Options +Indexes
</Directory>
</VirtualHost>

<VirtualHost 65.201.93.140:80>
DocumentRoot /usr/home/vdomain3/www/data
ServerName www.vdomain3.com
ServerAlias vdomain3.com *.vdomain3.com
ErrorLog "/usr/home/vdomain3/log/error.log"
TransferLog "/usr/home/vdomain3/log/access.log"
<Directory "/usr/home/vdomain3/www/data">
allow from all
Options +Indexes
Options ExecCGI 
</Directory>
</VirtualHost>

After configuring the Apache httpd.conf file, you will have to stop and restart the httpd daemon (the correct way to stop and start the Apache web server is to use the commands "/usr/local/sbin/apachectl stop" and "/usr/local/sbin/apachectl start"). Next, use a web browser and access each of the virtual domains that the ErrorLog and TransferLog tags were added to. SSH into the server or use the Webmin file manger and check the /usr/home/username/log directories and make sure they now contain the error.log and access.log files that should have been generated when the domains were accessed by a browser.

Webalizer configuration files for each virtual domain now need to be configured. Create a directory called /etc/webalizer. In this directory, use vi to create a file containing the following:

LogFile /usr/home/vdomain1/log/access.log
OutputDir /usr/home/vdomain1/www/data/stats
HostName www.vdomain1.com
HideReferrer vdomain1.com*

If a configuration file for our three example domains were created in /etc/webalizer, a directory listing would show:

vdomain1.conf
vdomain2.conf
vdomain3.conf

If it hasn't been done yet, an output directory (OutputDir) for Webalizer should be created in a directory of each virtual host, and it should be one that is accessible from the Internet (example: /usr/home/vdomain/www/data/stats). Using this directory structure for each virtual host, end users would access their Webalizer output by using a browser to go to: http://www.domain-name.com/stats.

If Webalizer has been installed, a configuration file for each virtual host has been created, and the proper directories, such as the Webalizer OutputDir, have been created, Webalizer can be tested by SSHing into the server and issuing the following command (assuming that the configuration file for our example of vdomain1 is being used):

/usr/local/bin/webalizer -c /etc/webalizer/vdomain1.conf

Which will give the following output:

Webalizer V2.01-10 (FreeBSD 6.0-RELEASE) English
Using logfile /usr/home/vdomain1/log/access.log (clf)
Creating output in /usr/home/vdomain1/www/data/stats
Hostname for reports is 'www.vdomain1.com'
Reading history file... webalizer.hist
Generating report for July 2006
Generating summary report
Saving history information...
91 records in 0.08 seconds

For whichever Webalizer domain configuration file was chosen to manually run our test with, use a web browser and go to the Webalizer output directory of that domain (example: /usr/home/vdomain/www/data/stats). If everything is configured properly, the Webalizer output data will be pulled up in the web browser.

If the test is successful, the only step left for a basic working setup of Webalizer, is to creat a cron job so that stats are generated at whatever interval is chosen (usually nightly or twice a day).

To create a cron job to automatically run Webalizer on each virtual host, use vi to create webalizer.sh in the /usr/sbin/ directory (issue a chmod 550 on the file). In the webalizer.sh file, enter the following:

#!/bin/sh
cd /etc/webalizer
for i in *
do /usr/local/bin/webalizer -c $i;
done 

Use Webmin to schedule a cron job to run at whatever interval is needed. The command used in the cron job will be:

/usr/sbin/webalizer.sh

 

Problem: webalizer: Command not found.

Solution: All of the documentation on Webalizer shows the used of the command: webalizer. However, on FreeBSD, the command will not be found. Use the following command in its place:

/usr/local/bin/webalizer


Note: Here is an example of output from Webalizer, configured with the above settings: Webalizer (basic)


Webalizer, with more advanced settings
:

By changing the Apache log tags, it is possible to get stats on referrers and type of browsers people are using to access your web site. The following tags will enable this function:

LogFormat "%h %l %u %t \"%r\" %>s %b" common
CustomLog /usr/home/network/log/access.log combined

Continuing to use our virtual hosts that have been used as examples up to this point, our virtual host configurations with the above tags would look like this: 

<VirtualHost 65.201.93.140:80>
DocumentRoot /usr/home/vdomain1/www/data
ServerName www.vdomain1.com
ServerAlias vdomain1.com *.vdomain1.com
ErrorLog "/usr/home/vdomain1/log/error.log"
TransferLog "/usr/home/vdomain1/log/access.log"
<Directory "/usr/home/vdomain1/www/data">
allow from all
Options +Indexes
</Directory>
</VirtualHost>

The above settings will add a referrer, a search string used in a search engine used to find the web site, and the user agent (browser) used to visit the web site.

Top 6 of 6 Total Referrers

#

Hits

Referrer

1

14

4.86%

http://www.inetwork-plus.com/

2

12

4.17%

http://www.inetwork-plus.com/stats/usage_200607.html

3

7

2.43%

http://www.inetwork-plus.com/stats/

4

6

2.08%

- (Direct Request)

5

6

2.08%

http://www.inetwork-plus.com/about.htm

6

2

0.69%

http://www.google.com/search

Top 1 of 1 Total Search Strings

#

Hits

Search String

1

2

100.00%

palestine TX networking

2

2

100.00%

palestine TX networking

3

2

100.00%

palestine Texas networking

4

2

100.00%

palestine Texas networking & computers

Top 2 of 2 Total User Agents

#

Hits

User Agent

1

34

11.81%

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.4) G

2

13

4.51%

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET

Usage Statistics for www.inetwork-plus.com - July 2006


Without having Apache performing HostnameLookups, or having Webalizer configured to perform DNS resolution, the country statistics, which keep totals on where visitors originate from, Webalizer will show 100% "Unresolved/Unknown"

Because doing domain name lookups on Apache can slow down a busy site, Apache recommends not enabling this feature. However, Webalizer, while it can do nameserver lookups, recommends that Apache handle the lookups. After researching the issue, it is recommend that if Apache is going to be used to do the lookups, that a utility, such as logresolve, should be used to do the lookups in a separate process. Since Webalizer can also do a similar function, Webalizer will be used to do DNS lookups in this example.

Since we have configured all of our log files to be stored in the directory of each individual virtual domain, we need to configure Webalizer to analyze the log file created by each individual virtual domain, perform a DNS lookup, and store the result of the lookups in a file, cache.db. This allows Webalizer to be used in a two step process. One to create the data in the cache.db database, and one to run Webalizer against this database in order to create its normal log file analysis.

In order to do this, add the following line to the webalizer.sh script configured in our original basic setup of webalizer:

for i in /usr/home/vdomain1/log/access.log; do
webazolver -N 20 -D /etc/cache.db $i !/bin/sh; done

For example, continuing to use our example scenario already established earlier, a webalizer.sh file configured for three virtual hosts would be:

#!/bin/sh
for i in /usr/home/vdomain1/log/access.log; do
webazolver -N 20 -D /etc/cache.db $i !/bin/sh; done
for i in /usr/home/vdomain2/log/access.log; do 
webazolver -N 20 -D /etc/cache.db $i !/bin/sh; done 
for i in /usr/home/vdomain3/log/access.log; do 
webazolver -N 20 -D /etc/cache.db $i !/bin/sh; done
cd /etc/webalizer
for i in *
do /usr/local/bin/webalizer -c $i -D /etc/cache.db; 
done 

The above would process each log file, resolve IP addresses to domain names, and place the data in /etc/cache.db. When every log file has been processed, it will then run each Webalizer configuration file, found in /etc/webalizer and process it using the DNS cache file /etc/cache.db.

After the webalizer.sh script file has been configured correctly, it can be tested by the command:

 /usr/sbin/webalizer.sh

If it works correctly, no further action is necessary, as the webalizer.sh script was already configured to run via a cron job in our basic setup of Webalizer. Properly configured, the advanced setup of Webalizer will add the following to the Webalizer output file:

Usage by Country for July 2006

 
Top 5 of 5 Total Countries
# Hits Files KBytes Country
1 198 63.46% 126 67.02% 1406 65.09% Unresolved/Unknown
2 96 30.77% 48 25.53% 619 28.66% US Commercial
3 14 4.49% 14 7.45% 85 3.94% Zimbabwe
4 2 0.64% 1 0.53% 25 1.16% US Educational
5 2 0.64% 1 0.53% 25 1.16% Seychelles

Generated by Webalizer Version 2.01

Usage Statistics for www.inetwork-plus.com - July 2006


Note: Here is an example of output from Webalizer, configured with the above, advanced settings: Webalizer (advanced)


Since DNS lookups are quite time consuming, it is recommended that the webazolver process be done very late at night, or other time, when server activity is at its lowest point.

Also, remember to update these key files whenever a new virtual host is added to the server:

Security issues:

Since owners of the hosted virtual domainsr may or may not want their stats accessible by the world, all stats directories, by default, should be configured to have password protection provided by an .htaccess file. If users want to delete this file and allow access, it should be at their discretion.