Tlog

Tlog is a terminal I/O recording and playback package suitable for implementing [centralized user session recording][session_recording].

Whereas most other similar packages write the recorded data to a file in their own format, or upload it to a custom server, tlog sends it to a logging service. Both the standard syslog and the journald interfaces are supported. The recorded data is [encoded][log_format] in JSON in a way which keeps it human-readable and searchable as much as possible.

The primary purpose of logging in JSON format is to eventually deliver the recorded data to a storage service such as [Elasticsearch][elasticsearch], where it can be searched and queried, and from where it can be played back.

Tlog contains three tools: tlog-rec for recording terminal I/O of programs or shells in general, tlog-rec-session for recording I/O of whole terminal sessions, with protection from recorded users, and tlog-play for playing back the recordings. You can run tlog-rec for testing or recording specific commands or shell sessions for yourself, or you can integrate it into another solution. Tlog-rec-session is intended to be a user's login shell. It puts itself between the actual user's shell and the terminal upon user login, logging everything that passes through. Lastly, tlog-play can playback recordings from Elasticsearch or from a file, made with either tlog-rec or tlog-rec-session. There is no difference in log format between tlog-rec and tlog-rec-session.

Building

Build dependencies are systemd, cURL, json-c, and libutempter, which development packages are systemd-devel, json-c-devel, libcurl-devel, and libutempter-devel on RPM-based distros, and pkg-config, libjson-c-dev, libsystemd-journal-dev/libsystemd-dev, libcurl-*-dev, and libutempter-dev on Debian-based distros.

To build from Git you'll need autoconf, automake and libtool packages. For creating RPM package rpm-build is also required.

If Systemd Journal support is not required, it can be disabled with configure's --disable-journal option, removing the systemd dependency as well.

Updating the system utmp and wtmp files can be enabled with the --enable-utempter configure option, utilizing the libutempter library.

If you'd like to build tlog from the Git source tree, you need to first generate the build system files:

autoreconf -i -f

After that, or if you're building a [release source tarball][releases], you need to follow the usual configure & make approach:

./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var && make

To generate a source tarball:

./configure --prefix=/usr --sysconfdir=/etc && make dist

From a source tarball you can build an SRPM package:

rpmbuild -ts <tarball>

Or an RPM package:

rpmbuild -tb <tarball>

Installing

You can use one of the [release binary RPM packages][releases] and install them with your favorite tool. The RPM package does all the necessary setup for you.

Otherwise, if you built tlog from source, you can install it with the usual make install:

sudo make install

If you are recording other user sessions, and don't want them to be able to affect the recording process, make sure you use tlog-rec-session and its executable is SUID/SGID to separate and dedicated user and group. It doesn't require running as root and will be safer with a regular, dedicated user.

You will also need to create the session lock directory /var/run/tlog and make it writable (only) for the user(s) tlog-rec-session runs as. On systems where (/var)/run is a tmpfs you will also need to make sure the session lock directory is recreated on the next boot. In that case, on systems with systemd you'll need to create a tmpfiles.d configuration file, and on init.d systems - a startup script, creating the directory for you.

Testing

You can test if session recording and playback work in general with a freshly installed tlog, by recording a session into a file with tlog-rec and then playing it back with tlog-play.

Usage

Recording to a file

To record into a file, execute tlog-rec on the command line as such:

tlog-rec --writer=file --file-path=tlog.log

Playing back from a file

Both during, and after the recording you can play the session back with tlog-play:

tlog-play --reader=file --file-path=tlog.log

Recording to Systemd Journal

To record into the Systemd Journal, execute tlog-rec as such:

tlog-rec --writer=journal

Along with the regular JSON log messages, when recording to Journal, tlog copies a few JSON fields to Journal fields (unless explicitly disabled) to aid searching and extracting (parts of) particular recordings:

• TLOG_USER - the user the recording was started as (user in JSON),
• TLOG_SESSION - the audit session ID of the recording process (session in JSON),
• TLOG_REC - host-unique recording ID (rec in JSON),
• TLOG_ID - log message ID within the recording (id in JSON).

Playing back from Systemd Journal

In general, selecting Journal log entries for playback is done using Journal matches and timestamp limits, with -M/--journal-match, -S/--journal-since, and -U/--journal-until options.

In practice however, playback from Journal is usually done with a single match against the TLOG_REC Journal field. The TLOG_REC field contains a copy of the rec field from the logged JSON data, which is a host-unique ID of the recording. For example, to playback a recording which contains this message (output with journalctl -o verbose and abbreviated):

Mon 2018-01-22 10:51:48.463904 EET [s=87ea0a3f655a48cd80d7f49053860806;...
    _AUDIT_LOGINUID=1000
    _UID=1000
    _GID=1000
    _AUDIT_SESSION=2
    _BOOT_ID=12ca5b356065453fb50adfe57007658a
    _MACHINE_ID=2d8d017e2b1144cbbdd049a8a997911b
    _HOSTNAME=bard
    PRIORITY=6
    _TRANSPORT=journal
    _SYSTEMD_OWNER_UID=1000
    TLOG_REC=12ca5b356065453fb50adfe57007658a-306a-26f2910
    TLOG_USER=nkondras
    TLOG_SESSION=2
    TLOG_ID=1
    MESSAGE={"ver":"2.3","host":"bard","rec":"12ca5b356065453fb50adfe57007658a-306a-26f2910",...
    SYSLOG_IDENTIFIER=tlog-rec
    _PID=12394
    _COMM=tlog-rec
    _SOURCE_REALTIME_TIMESTAMP=1516611108463904

you can take the ID either from the TLOG_REC field value directly, or from the MESSAGE field (from the JSON rec field). You can then playback the whole recording like this:

tlog-play -r journal -M TLOG_REC=12ca5b356065453fb50adfe57007658a-306a-26f2910

When compiled with Systemd >= 245 it is possible to read log entries from a specific Journal namespace using parameters -N/--journal-namespace.

Playing back ongoing recordings

By default, once tlog-play reaches the last message a recording has so far, it exits. However, it can be made to poll for new messages appearing with the -f/--follow option, which is useful for playing back ongoing recordings.

Playback controls

Tlog-play accepts several command-line options affecting playback, including -s/--speed for setting playback speed multiplier, -g/--goto for specifying the location the playback should be fast-forwarded to, and -p/--paused for starting playback in paused state.

Several control keys are also recognized during playback:

• SPACE or p for pause/resume,
• { and } for halving and doubling the playback speed,
• . for stepping through the recording (on pause or during playback)
• G for fast-forwarding to the end of the recording (useful with --follow), or to the specified timestamp (see tlog-play(8) for details),
• and q for quitting playback.

Rate-limiting recording

Both tlog-rec and tlog-rec-session can be setup to limit the rate at which recording's messages are logged. Tlog-rec accepts three options: --limit-rate=NUMBER, --limit-burst=NUMBER, and --limit-action=STRING, which specify rate limit in bytes per second, burst limit in bytes, and the limit action (pass/delay/drop), respectively. The same parameters can be changed using limit.rate, limit.burst, and limit.action configuration parameters for both tlog-rec and tlog-rec-session.

The default pass limit action lets all the messages through unhindered, effectively disabling rate-limiting. You can throttle logging, and slow down the user's terminal I/O using the delay limit action. Finally, you can simply drop the captured I/O, going above the rate and burst limits, using the drop action. See tlog-rec(8), tlog-rec.conf(5), and tlog-rec-session.conf(5) for details.

Playing back partial recordings

By default tlog-play will terminate playback, if it notices out-of-order or missing log messages. However, it is possible to make it ignore missing messages with the --lax option for when you need to playback a partial or a corrupted recording.

Recording sessions of a user

Change the shell of the user to be recorded to tlog-rec-session:

sudo usermod -s /usr/bin/tlog-rec-session <user>

Login as the user on a text terminal. By default the recorded terminal data will be delivered to Journal (if tlog was built with Journal support) or to syslog with facility "authpriv". In both cases default priority will be "info". It will appear in Journal, if you use journald, or in /var/log/auth.log on Debian-based systems, and in /var/log/secure on Fedora and derived systems.

Customize tlog-rec-session configuration in /etc/tlog/tlog-rec-session.conf as necessary (see tlog-rec-session.conf(5) for details).

Locale configuration issue on Fedora and RHEL

Fedora and RHEL