---
layout: docs
title: Logfile Cache
breadcrumb: [ "advanced.html", "Advanced" ]
---


== Logfile Cache
Accessing logfiles can be time and memory consuming, especially when having
remote sites connected to Thruk. Livestatus caches logfile data used for
queries. This increases the size of the naemon process, slows down the
monitoring and increases the overall load on your monitoring host. Therefor
Thruk may use a logfile cache to increase performance of logfile access and
reports.

For maximum performance, the Logfile Cache Database should be installed on the
same server as Thruk itself (or at least in the same network).

The idea is to store complete log data for about half a year in the cache. Then
up to 2 years of compacted logfiles to generate sla reports.


=== Configuration

[TIP]
Starting with OMD-Labs 2.40 you can simply run `omd config set MYSQL on; omd config set THRUK_LOGCACHE on` to enable the logcache.

You will need to specify your mysql connection configuration in your
thruk_local.conf. Make sure the user and database exists. The tables will then
be created automatically.

thruk_local.conf:
-------
  logcache = mysql://username:password@localhost:3306/thruk_logs
-------

[TIP]
.create mysql user and tables
=======
 #> mysql -u root -p
 mysql> CREATE USER 'thruk'@'localhost' IDENTIFIED BY '****';
 mysql> CREATE DATABASE IF NOT EXISTS `thruk_logs`;
 mysql> GRANT ALL PRIVILEGES ON `thruk_logs`.* TO 'thruk'@'localhost';
=======


=== Actions

==== Import
It's a good idea to import the logfiles after enabling the
logfilecache:

Initial import:
-------
    %> thruk -a logcache import --local -v
    ...
    OK - imported 20374 log items from 1 site successfully in 4.94s (4122/s)
-------

Be careful, the 'logcacheimport' deletes all current cached data, so only
do that once.

Depending on the size, this may take a while.

==== Update

Logfile Cache Update
-------
    %> thruk -a logcache update -v
    OK - imported 3 log items from 3 sites successfully in 0.13s (23/s)
-------

The logcacheupdate imports the delta since the last update. That's pretty
fast and done automatically whenever you access logfiles via Thruk. However,
if you access the eventlog page only once in a while, it would be a good idea
to update the logcache periodically by a cronjob.


==== Import from Logfile Archive Files
The MySQL logcache supports import from files too. Duplicates are not imported
twice, so you can do that safely as often as you like.

-------
    %> thruk -a logcache update --local /var/log/naemon/archive/naemon-09-07-2012-00.log
    OK - imported 2 log items from 1 site successfully in 0.01s (247/s)
-------


==== Authentication Update

The cache needs authentication information which has to be updated whenever
contacts have changed.

Update logfile caches authentication information:
-------
    %> thruk -a logcache authupdate
    OK - updated 71 log items from 3 sites successfully in 0.25s (286/s)
-------


==== Cleanup
You may want to remove old logfile entries from time to time. This is possible
with the logcacheclean command. Optionally you can define the number of days
which should be kept in the cache.

-------
    %> thruk -a logcache clean -v
    OK - removed 18046 log items from 1 site successfully in 0.08s (218816/s)
-------

==== Compact
Compact removes duplicate information which is not required for sla reporting.
Notifications will also kept.

-------
    %> thruk -a logcache compact -v
    OK - compacted 18046 log items from 1 site successfully in 0.08s (218816/s)
-------


==== Statistics

There is also a command to check the current usage statistics of your
logfile cache:

-------
    %> thruk -a logcache stats
    Backend              Index Size      Data Size       Items
    Devel                 14.4 MB        136.4 MB       238104
-------


=== Example Cronjobs
Using the logfile cache needs some extra maintenance to work smoothly, the
following cronjobs should cover that. When not using OMD, you will have to
adjust the paths.

-------
  # update logcache data
  */5 * * * * $OMD_ROOT/bin/thruk -a logcache update     >>$OMD_ROOT/var/log/thruk.log 2>&1

  # update authentication data in our logcache
  0   2 * * * $OMD_ROOT/bin/thruk -a logcache authupdate >>$OMD_ROOT/var/log/thruk.log 2>&1

  # optimize and repair tables
  0  20 * * * $OMD_ROOT/bin/thruk -a logcache optimize   >>$OMD_ROOT/var/log/thruk.log 2>&1

  # clean logfiles from cache after default duration (thruk.conf: logcache_clean_duration)
  0   1 * * * $OMD_ROOT/bin/thruk -a logcache clean      >>$OMD_ROOT/var/log/thruk.log 2>&1

  # compact logfiles from cache after default duration (thruk.conf: logcache_compact_duration)
  0   1 * * * $OMD_ROOT/bin/thruk -a logcache compact    >>$OMD_ROOT/var/log/thruk.log 2>&1
-------


=== Icinga 2
When using Icinga 2 it is highly recommended to use the logcache with a file
import because the Livestatus API returns wrong results for the logs table in
Icinga 2. If the file import is not possible, there is an experimental ido
import available.

==== File Import

For file imports the following settings are recommended (OMD-Labs):

etc/thruk/thruk_local.d/logcache.conf:
-------
  logcache = mysql://root@tmp/run/mysqld/mysqld.sock/thruk_logs
  logcache_delta_updates = 0
  report_update_logcache = 0
-------

This disables on-demand updates which won't work.


Then use a slightly changed crontab which imports the logfiles directly from the
compat logfiles.

etc/cron.d/thruk_logcache
-------
  # update logcache data
  * * * * * $OMD_ROOT/bin/thruk -a logcache update var/icinga2/log/icinga2/compat/icinga.log >>$OMD_ROOT/var/log/thruk.log 2>&1
  5 0 * * * $OMD_ROOT/bin/thruk -a logcache update $(ls -1tr var/icinga2/log/icinga2/compat/archives/*.log) >>$OMD_ROOT/var/log/thruk.log 2>&1

  # update authentication data in our logcache
  0   2 * * * $OMD_ROOT/bin/thruk -a logcache authupdate >>$OMD_ROOT/var/log/thruk.log 2>&1

  # optimize and repair tables
  0  20 * * * $OMD_ROOT/bin/thruk -a logcache optimize   >>$OMD_ROOT/var/log/thruk.log 2>&1

  # clean logfiles from cache after default duration (thruk.conf: logcache_clean_duration)
  0   1 * * * $OMD_ROOT/bin/thruk -a logcache clean      >>$OMD_ROOT/var/log/thruk.log 2>&1

  # compact logfiles from cache after default duration (thruk.conf: logcache_compact_duration)
  0   1 * * * $OMD_ROOT/bin/thruk -a logcache compact    >>$OMD_ROOT/var/log/thruk.log 2>&1
-------


==== IDO Import
{% include new_since.ad version="2.22" %}
For clustered icinga2 setups were the compat-log feature will result in incorrect
timestamps you might want to use the experimental ido import script.

However there a few known issues:

  - no support for timeperiods, reports will be 24x7 only
  - no initial states which might result in unknown states for sla reports
  - no command names for notifications

thruk_local.conf
-------
logcache_fetchlogs_command = IDO_DB_HOST=127.0.0.1 IDO_DB_PORT=3306 IDO_DB_USER=root IDO_DB_PW=root IDO_DB_NAME=icinga /usr/share/thruk/support/icinga2_ido_fetchlogs.sh mysql
-------

if using OMD, change the path to

thruk_local.conf
-------
logcache_fetchlogs_command = IDO_DB_HOST=127.0.0.1 IDO_DB_PORT=3306 IDO_DB_USER=root IDO_DB_PW=root IDO_DB_NAME=icinga ./share/thruk/support/icinga2_ido_fetchlogs.sh mysql
-------

For postgres database replace the literal 'mysql' with 'postgres' in the command above.
