Reimplementation of the check_logfiles Nagios plugin

Nagios check_logfiles

If you're familiar with Nagios plugins, you probably should be aware of its check_logfiles one. It allows to scan files (generally log files generated by UNIX daemons or Windows services) for different patterns. Each time a pattern is matched, an external script is called, along with arguments like regex captures groups.

The plug-in uses configuration files, which are plain Perl language regular files. These are imported by the check_logfiles script which uses them internally.

You can find the check_logfiles plugin here: https://labs.consol.de/nagios/check_logfiles/

Implementation in Rust

While Perl is one of the fastest interpreted language, nothing can beat a compiled executable, specially when developed with a non-garbage collected one like C, C++ or Rust. When dealing with lots of servers in a professional environment, execution speed, memory footprint and scalability are paramount.

Rust is a relatively new system's programming language, providing speed, and security at compile time. For an introduction to Rust, please refer to https://www.rust-lang.org/ .

This reimplementation in Rust aims at solving original check_logfiles drawbacks:

• straightforward distribution: a single executable is needed. Using the musl static binary, no need for a specific verison of libc
• enhanced portability: same behaviour between Windows, Linux or BSD-like operating systems
• lightning speed: with an execution speed comparable to C or C++, execution time is not a hurdle
• ability to either call an external script or send JSON data coming from logfile to a TCP stream or a UNIX domain socket
• standard configuration file format: opposite to the original check_logfiles with uses non-standard configuration files (regular Perl files containing Perl variables), this implementation uses the YAML format for its configuration files. YAML is best suited comparing to JSON or XML because there's no need to escape chars for regexes expressions
• versatility: coupled with Jinja2-like well-known templates, you can imagine lots of possibilities to manage configuration files in a professionnal environment
• power: it will take into account not only regular log files, but also list of files command from a shell command or a script
• no need for a decompression binary: logfiles are gunzipped out of the box. Supported formats: gzip (extension: .gz), bzip2 (extension: .bz2), xz (extension: .xz)
• search for current or archived log files
• manage log rotations
• UTF-8-ready by default

Caveat: even though the current regex Rust crate is providing fast regexes checks, it doesn't currently support lookahead or lookbehind patterns out of the box.

Releases

Current release is 0.8 and currently still in a testing phase.

Principle of operation

The clf executable processing is simple. After reading the YAML configuration file passed to the command line, it reads each logfile (or a list of logfiles provided by an external command or script) and tests each line against the defined regexes. If a match is found, it triggers an external script, either by spawning a new process and providing a set of environment variables to this process (and optionnally updating the PATH or Path variable, depending on the OS). Or by establishing a new socket connection to a remote address and port configured in the configuration file, and sending a JSON data structure with a set of variables coming from the search. Or even to a UNIX domain socket with the same principle.

The plugin output and exit code is depending on what is found in the provided logfiles.

Format of the YAML configuration file

The current format of the configuration file defines where and what to search is a standard YAML format.

Following is a list of current tags defined in the configuration file with a description of each tag:

---
# a list of global options, valid for all searches.
global:
  # a list of ':'-separated list of paths (UNIX) or ';'-separated (Windows). If provided, the script is
  # searched within those directories.
  # Defaults to '/usr/sbin:/usr/bin:/sbin:/bin' or 'C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;' if not provided, depending on the 
  # platform.
  script_path: "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

  # a path & file name where clf is keeping its runtime information. This is used to restart the search from the last
  # offset reached from the last clf run. If not provided, the file is the config file path with .json extension
  # could be overriden by the -d command line argument
  snapshot_file: /tmp/snapshot.json

  # retention time for tags in seconds. Defaults to 7 days
  snapshot_retention: 3600

  # a list of user variables, if any. Provided as-is to the callback (no CLF_ prefix)
  vars:
    first_name: Al
    last_name: Pacino
    city: 'Los Angeles'
    profession: actor

  # a list of executables or scripts started before searching into logfiles
  prescript:
    # command to run with its arguments
    - command: 
        - ./tests/integration/callbacks/echodomain.py
        - /tmp/echodomain.txt
        - /tmp/echodomain.sock

      # sleep current thread before continuing. In ms, defaults to 10
      timeout: 1000
  
      # async=true means clf will not wait for prescript succesfull execution
      async: true
  
      # stops clf processing if prescript return code is non 0
      exit_on_error: true

  # a command run at the end of logfiles processing. The list of pids from prescripts
  # is sent as arguments to this command
  postscript:
    command: ['./tests/integration/callbacks/kill.py']    
    timeout: 1000


# a list of logfiles & tags, to search for patterns. This is either a list of logfiles, or a command giving back a list of 
# files to search for.
searches:
  # logfile features
  - logfile: 
      # logfile path
      path: ./examples/logfiles/access_simple.log

      # format could be plain or JSON (JOSN to be developed)
      format: plain

      # lines matching the regex will be ignored
      exclude: ^#

      # specify logfile archive path and extension when rotated
      archive:
        # directory of archive. If not specified, current logfile directory
        dir: /tmp

        # archive extension
        extension: gz

      # what to report when a logfile is not found. Could be: critical, warning, unknown
      logfilemissing: critical

      # to determine whether a logfile has been rotated, inode & dev numbers might not be faithful. This is 
      # a buffer size which is used to calculate a CRC64 hash in case of inodes & devs are equal. Defaults to 4096
      hash_window: 2048


    # list of tags to refer to
    tags: 

      # tag name
      - name: http_access_get_or_post

        # set it to false to skip logfile processing. Defaults to true
        process: true

        # a list of comma-separated options to manage the search. See below for a list of options
        options: "runcallback"

        # a script or command to be called, every time a hit is found.
        callback: 
          script: ./tests/integration/callbacks/echovars.py
          args: 
            - /tmp/echovars.txt
            - arg2
            - arg3
          # a timeout specific for each callback kind (default: 5s)
          timeout: 10

        # list of patterns to match
        patterns:
          warning:
            # a list of regexes whic are considered as an error
            regexes:
              - 'GET\s+([/\w]+_logo\.jpg)'
              - 'POST\s+([/\w\.]+)\s+HTTP/1\.1"\s+(?P<code>\d+)\s+(?P<length>\d+)'
            # if one of the exception regexes match, this is not considered as an error
            exceptions:
              - 'Firefox/63.0'
              - 'AppleWebKit/537\.36'


      # another tag for the same logfile
      - name: http_access_images
        options: "runcallback"
        callback: 
          domain: /tmp/echodomain.sock
          args: ['/tmp/echodomain.txt', 'arg2', 'arg3']
        patterns:
          critical:
            regexes: 
              - 'GET\s+([/\w]+)\s+HTTP/1\.1"\s+(?P<code>\d+)\s+(?P<length>\d+)'
              - 'POST\s+([/\w\.]+)\s+HTTP/1\.1"\s+(?P<code>\d+)\s+(?P<length>\d+)'
            exceptions:
              - 'AppleWebKit/537\.36'

     
  # this time, a list of files is given by executing this command and capturing its output.
  - logfile:
      list: ['find','/var/log', "-maxdepth", "2", "-type", "f", "-name", "[a-d]*.log"]
    tags: 
      - name: all_logs
        options: "runcallback"
        callback: 
          domain: /tmp/echodomain.sock
          args: ['/tmp/echodomain.txt', 'arg2', 'arg3']
        patterns:
          critical:
            regexes: 
              - 'error'

The way clf searches in logfiles is controlled, for each tag, by a list of comma-separated options. Some are only boolean options, soem others require to be set using an equality:

option | description --- | --- runcallback | if set, the defined callback will be call for each line where a critical or warning pattern matches rewind | if set, clf will read the considered logfile from the beginning, bypassing any offset recorded in the snapshot file fastforward | move to the end of the file, don't call any callback, if no snapshot data is found for the logfile criticalthreshold=n | when set to a 8-byte positive integer value, it means that critical errors will not be triggered unless this threshold is reached warningthreshold=n | when set to a 8-byte positive integer value, it means that warning errors will not be triggered unless this threshold is