Back
Featured image of post Nginx analytics with Matomo on FreeBSD

Nginx analytics with Matomo on FreeBSD

Websites analytics based on logs using Matomo and Nginx on FreeBSD

Zabbix is an enterprise open source monitoring solution for networks and applications.

Nginx configuration

New log format

Edit the nginx configuration :

vim /usr/local/etc/nginx/nginx.conf

Add the following in the http section :

http {
	...
    log_format  matomo    '{"ip": "$remote_addr",'
                          '"host": "$host",'
                          '"path": "$request_uri",'
                          '"status": "$status",'
                          '"referrer": "$http_referer",'
                          '"user_agent": "$http_user_agent",'
                          '"length": $bytes_sent,'
                          '"generation_time_milli": $request_time,'
                          '"date": "$time_iso8601"}';
	...
}

Then, to use the log format, proceed as below :

    server {
		...
        access_log                      /var/log/nginx/access.log matomo;
		...
	}

Restart nginx

service nginx restart

Then, you should have logs in a json format. You can check :

tail /var/log/nginx/access.log
You should have :
{"ip": "X.X.X.X","host": "domain.tld","path": "/piwik.php","status": "200","referrer": "-","user_agent": "Matomo/LogImport","length": 331,"generation_time_milli": 3.119,"date": "2021-04-19T16:06:05+02:00"}

System configuration

Log tail

To avoid to insert duplicated data, we’ll use logtail for log parsing. Logtail is not available in FreeBSD, but installing logcheck will install logtail automatically ! Install logcheck :

pkg install logcheck

As logtail will be used by the www user on the access log, let’s create the file :

touch /var/log/nginx/access.log.offset
chown www:www /var/log/nginx/access.log.offset

Matomo token

After a few bugs, it’s more reliable to create a token to be used for authentication.

Go to : Matomo configuration > Personnal > Security, then create a token as below :

Matomo token for import_logs.py
Matomo token for import_logs.py

Cron job

In order to feed matomo, we’ll use a cron job.

vim /etc/cron.d/matomo.conf

Add the following content :

# Logs analysis
* * * * * www /usr/local/sbin/logtail /var/log/nginx/access.log | /usr/local/bin/python/path/to/matomo/misc/log-analytics/import_logs.py --url=https://matomo.domain.tld --token-auth=MyToken --enable-http-errors --enable-http-redirects --log-format-name=nginx_json -
# Reports archive
5 * * * * www /usr/local/bin/php /path/to/matomo/onsole core:archive --url=https://matomo.domain.tld 

Note : run run a command with an other user on Freebsd :

su -m www -c '/usr/local/sbin/logtail /var/log/nginx/access.log | /usr/local/bin/python/path/to/matomo/misc/log-analytics/import_logs.py --url=https://matomo.domain.tld --token-auth=MyToken --enable-http-errors --enable-http-redirects --log-format-name=nginx_json -'

The output should be something like :

0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log (stdin)...
65 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
65 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)

Logs import summary
-------------------

    63 requests imported successfully
    0 requests were downloads
    64 requests ignored:
        0 HTTP errors
        0 HTTP redirects
        0 invalid log lines
        0 filtered log lines
        62 requests did not match any known site
        0 requests did not match any --hostname
        0 requests done by bots, search engines...
        2 requests to static resources (css, js, images, ico, ttf...)
        0 requests to file downloads did not match any --download-extensions

Website import summary
----------------------

    63 requests imported to 1 sites
        1 sites already existed
        0 sites were created:

    1 distinct hostnames did not match any existing site:
            matomo.domain.tld

        TIPs:
         - if one of these hosts is an alias host for one of the websites
           in Matomo, you can add this host as an "Alias URL" in Settings > Websites.
         - use --add-sites-new-hosts if you wish to automatically create
           one website for each of these hosts in Matomo rather than discarding
           these requests.
         - use --idsite-fallback to force all these log lines with a new hostname
           to be recorded in a specific idsite (for example for troubleshooting/visualizing the data)
         - use --idsite to force all lines in the specified log files
           to be all recorded in the specified idsite
         - or you can also manually create a new Website in Matomo with the URL set to this hostname


Performance summary
-------------------

    Total time: 2 seconds
    Requests imported per second: 22.72 requests per second

Processing your log data
------------------------

    In order for your logs to be processed by Matomo, you may need to run the following command:
     ./console core:archive --force-all-websites --url='https://matomo.domain.tld'

All done, you should now be able to feed your matomo plateform from your Nginx logs on a FreeBSD server.

Built with Hugo
Theme Stack designed by Jimmy