Thruk - Logfile Cache

Logfile Cache

Accessing logfiles can be time and memory consuming, especially when having remote sites connected to Thruk. Livestatus caches logfile data used for queries. This increases the size of the naemon process, slows down the monitoring and increases the overall load on your monitoring host. Therefor Thruk may use a logfile cache to increase performance of logfile access and reports.

For maximum performance, the Logfile Cache Database should be installed on the same server as Thruk itself (or at least in the same network).

The idea is to store complete log data for about half a year in the cache. Then up to 2 years of compacted logfiles to generate sla reports.

Configuration

Starting with OMD-Labs 2.40 you can simply run omd config set MYSQL on; omd config set THRUK_LOGCACHE on to enable the logcache.

You will need to specify your mysql connection configuration in your thruk_local.conf. Make sure the user and database exists. The tables will then be created automatically.

thruk_local.conf:

  logcache = mysql://username:password@localhost:3306/thruk_logs

create mysql user and tables

#> mysql -u root -p
mysql> CREATE USER 'thruk'@'localhost' IDENTIFIED BY '****';
mysql> CREATE DATABASE IF NOT EXISTS `thruk_logs`;
mysql> GRANT ALL PRIVILEGES ON `thruk_logs`.* TO 'thruk'@'localhost';

Actions

Import

It’s a good idea to import the logfiles after enabling the logfilecache:

Initial import:

    %> thruk -a logcache import --local -v
    ...
    OK - imported 20374 log items from 1 site successfully in 4.94s (4122/s)

Be careful, the 'logcacheimport' deletes all current cached data, so only do that once.

Depending on the size, this may take a while.

Update

Logfile Cache Update

    %> thruk -a logcache update -v
    OK - imported 3 log items from 3 sites successfully in 0.13s (23/s)

The logcacheupdate imports the delta since the last update. That’s pretty fast and done automatically whenever you access logfiles via Thruk. However, if you access the eventlog page only once in a while, it would be a good idea to update the logcache periodically by a cronjob.

Import from Logfile Archive Files

The MySQL logcache supports import from files too. Duplicates are not imported twice, so you can do that safely as often as you like.

    %> thruk -a logcache update --local /var/log/naemon/archive/naemon-09-07-2012-00.log
    OK - imported 2 log items from 1 site successfully in 0.01s (247/s)

Authentication Update

The cache needs authentication information which has to be updated whenever contacts have changed.

Update logfile caches authentication information:

    %> thruk -a logcache authupdate
    OK - updated 71 log items from 3 sites successfully in 0.25s (286/s)

Cleanup

You may want to remove old logfile entries from time to time. This is possible with the logcacheclean command. Optionally you can define the number of days which should be kept in the cache.

    %> thruk -a logcache clean -v
    OK - removed 18046 log items from 1 site successfully in 0.08s (218816/s)

Compact

Compact removes duplicate information which is not required for sla reporting. Notifications will also kept.

    %> thruk -a logcache compact -v
    OK - compacted 18046 log items from 1 site successfully in 0.08s (218816/s)

Statistics

There is also a command to check the current usage statistics of your logfile cache:

    %> thruk -a logcache stats
    Backend              Index Size      Data Size       Items
    Devel                 14.4 MB        136.4 MB       238104

Example Cronjobs

Using the logfile cache needs some extra maintenance to work smoothly, the following cronjobs should cover that. When not using OMD, you will have to adjust the paths.

  # update logcache data
  */5 * * * * $OMD_ROOT/bin/thruk -a logcache update     >>$OMD_ROOT/var/log/thruk.log 2>&1

  # update authentication data in our logcache
  0   2 * * * $OMD_ROOT/bin/thruk -a logcache authupdate >>$OMD_ROOT/var/log/thruk.log 2>&1

  # optimize and repair tables
  0  20 * * * $OMD_ROOT/bin/thruk -a logcache optimize   >>$OMD_ROOT/var/log/thruk.log 2>&1

  # clean logfiles from cache after default duration (thruk.conf: logcache_clean_duration)
  0   1 * * * $OMD_ROOT/bin/thruk -a logcache clean      >>$OMD_ROOT/var/log/thruk.log 2>&1

  # compact logfiles from cache after default duration (thruk.conf: logcache_compact_duration)
  0   1 * * * $OMD_ROOT/bin/thruk -a logcache compact    >>$OMD_ROOT/var/log/thruk.log 2>&1

Icinga 2

When using Icinga 2 it is highly recommended to use the logcache with a file import because the Livestatus API returns wrong results for the logs table in Icinga 2. If the file import is not possible, there is an experimental ido import available.

File Import

For file imports the following settings are recommended (OMD-Labs):

etc/thruk/thruk_local.d/logcache.conf:

  logcache = mysql://root@tmp/run/mysqld/mysqld.sock/thruk_logs
  logcache_delta_updates = 0
  report_update_logcache = 0

This disables on-demand updates which won’t work.

Then use a slightly changed crontab which imports the logfiles directly from the compat logfiles.

etc/cron.d/thruk_logcache

  # update logcache data
  * * * * * $OMD_ROOT/bin/thruk -a logcache update var/icinga2/log/icinga2/compat/icinga.log >>$OMD_ROOT/var/log/thruk.log 2>&1
  5 0 * * * $OMD_ROOT/bin/thruk -a logcache update $(ls -1tr var/icinga2/log/icinga2/compat/archives/*.log) >>$OMD_ROOT/var/log/thruk.log 2>&1

  # update authentication data in our logcache
  0   2 * * * $OMD_ROOT/bin/thruk -a logcache authupdate >>$OMD_ROOT/var/log/thruk.log 2>&1

  # optimize and repair tables
  0  20 * * * $OMD_ROOT/bin/thruk -a logcache optimize   >>$OMD_ROOT/var/log/thruk.log 2>&1

  # clean logfiles from cache after default duration (thruk.conf: logcache_clean_duration)
  0   1 * * * $OMD_ROOT/bin/thruk -a logcache clean      >>$OMD_ROOT/var/log/thruk.log 2>&1

  # compact logfiles from cache after default duration (thruk.conf: logcache_compact_duration)
  0   1 * * * $OMD_ROOT/bin/thruk -a logcache compact    >>$OMD_ROOT/var/log/thruk.log 2>&1

IDO Import

new in release v2.22

For clustered icinga2 setups were the compat-log feature will result in incorrect timestamps you might want to use the experimental ido import script.

However there a few known issues:

no support for timeperiods, reports will be 24x7 only
no initial states which might result in unknown states for sla reports
no command names for notifications

thruk_local.conf

logcache_fetchlogs_command = IDO_DB_HOST=127.0.0.1 IDO_DB_PORT=3306 IDO_DB_USER=root IDO_DB_PW=root IDO_DB_NAME=icinga /usr/share/thruk/support/icinga2_ido_fetchlogs.sh mysql

if using OMD, change the path to

thruk_local.conf

logcache_fetchlogs_command = IDO_DB_HOST=127.0.0.1 IDO_DB_PORT=3306 IDO_DB_USER=root IDO_DB_PW=root IDO_DB_NAME=icinga ./share/thruk/support/icinga2_ido_fetchlogs.sh mysql

For postgres database replace the literal 'mysql' with 'postgres' in the command above.