Introduction

Snebu -- the Simple Network Backup Utility, Systems Nominal, Everything's Backed Up

Snebu is an efficient, incremental, snapshot-style, de-duplicating, compressing, encrypting, database-backed, multi-host, client/server, simple, secure, disk-based backup system for Unix / Linux systems.

Super-quick setup

• After creating or editing /etc/snebu.conf and /etc/snebu-client.conf as specified below...

• And, if using encryption, generate a key tarcrypt genkey -f myhost.key

• If backing up to a local resource (locally attached storage):

  snebu-client backup -v -k myhost.key /home/mydir
  

• If backing up to a remote server:

  snebu-client backup -v --remote-server central-backup-server -k myhost.key \
    -k corporate-skeleton.key
  

• Or, if in a larger environment and you want to centrally manager backups from the backup server, write a script on the server and schedule it via cron with commands such as:

  snebu-client backup --remote-client host1.example.com -k key1.key -k corp.key
  snebu-client backup --remote-client host2.example.com -k key2.key -k corp.key
  

The optional -k parameters above specify encryption keys. These key files contain an RSA key pair, with the private key encrypted. The key data is stored on the backup server, and during a restore operation you will be prompted for the passphrase protecting the private key. The key file does not need to be present on the target client during a restore operation.

You can specify more than one key, in which case the passphrase for any of the keys can be used during restore.

If you don't specifiy a directory to back up, default include/exclude lists are processesed as specified in /etc/snebu-client.conf.

See the quick start guide below for more details. As you can see, you can back up a single host to some attached media, or to a remote host (even something simple as a Rapsberry Pi). Or you can

Features

• Efficient
  Incremental -- transfers only modified data across the network.
  Snapshot-style -- Each backup represents the complete state of the system.
  De-duplicating -- File-level de-duplication and compression, including de-dup across multiple hosts.

• Compresed/encrypted -- Uses built-in lzo based encryption, with optional RSA public key based encryption. With public key encryption, you don't have to be concerned with leaving a plain text encryption key on the client, or losing the encryption key. The encryption also utilizes an HMAC key to provide authenticity of the backup, and to allow for de-duplication of encrypted backups (including across multiple hosts, as long as the same key file is used on that group of hosts).

• Database-backed -- Metadata information is stored in an SQLite database.

• Multi-host -- Multiple systems can be backed up to a central target.

• Client-server -- Hosts can either push backups to a central backup server, or the server can pull backups from individual clients.

• Simple -- System consists of a single compiled binary, snebu, plus a front-end client, snebu-client written in Bash shell script. If used with multiple hosts, communications all happen over ssh, and there are no agent programs that need to be installed on remote hosts. Backed up files are stored in a "vault" directory, with the metadata in a light weight SQLite database file. (SQLite requires no database server processes, all functionality is built into the SQLite library linked against the main snebu binary).

• Secure -- Backup server can pull backups from hosts, or clients can push backups to the central backup server using minimal permission accounts (i.e., you can configure an account to be able to submit, but not delete backups, or restrict restores to only specific backup sets). Encryption is also provided by a separate isolated process, tarcrypt, which acts as a filter for tar files, compressing/encrypting the data contents while allowing the metadata to be processed and indexed by the backup server database.

Audience

Snebu is intended primarily for Linux based systems, or any Unix style system with GNU tar, GNU find, and ssh. Also works well under Windows with Cygwin.

Other Benefits

• Easy to set up
  There system consists of two programs:
  snebu-client - This is a simple shell script for interacting with the backend process, along with gathering the files to include in a given backup set.
  snebu -- This is the backend process which stores backup files in a disk-based vault, maintaining metadata about the files and backup sets in an SQLite-based backup catalog.

  There are no complicated daemons to configure, or heavy database systems that need to be installed. All the core logic is self-contained in the snebu binary.

• Suitable for backing up a single system to an attached external disk, backing up to a separate network server, or backing up a number of hosts with backups initiated from the central backup server.

• No client software installation required. If backups are initiated from a central backup server, then communications to the clients are preformed over SSH. The only requirements are that the backup account on the server is permitted appropriate access to the client (via ssh keys), and the client have the minimal versions of GNU find and tar installed (this is only a concern with ancient Linux installations, or Unix systems without GNU tools installed).

• Works similar to rsync-based backup tools (by providing snapshot incremental-forever backups), but provides compression and a self-contained database-backed (SQLite) backup catalog. One of the many benefits of having a separate backup catalog is this allows the backend to execute under a regular (non-root) utility account.

• File-level de-duplication is supported, including de-duplication of files across backup sets, and across multiple backup clients.

• Built using standard tools and protocols. File metadata gathering is provided by the "find" command (GNU find is required), files are transfered using "tar" format, the backup catalog is in an SQLite database file, and files are stored in the disk-based vault compressed with the LZO library, and compatible with the "lzop" compression program.

Quick-start guide

Concepts

The Snebu backup system is composed of a backend process, snebu, and a front-end client called snebu-client. The client gathers a list of files to back up (using the find command), sends this list to the backend, and gets back a list of files that are needed to complete an incremental backup. These files are then sent to the backend using the Unix tar format.

Backups are stored in backup sets. A backup set is uniquely identified by the backup name (typically the host name getting backed up), and a datestamp in the form of a Unix time_t value (essentially, a count of the number of seconds since Jan 1, 1970). This value functions as a serial number, and is used by the expiration process for purging old backups.

Each backup set has a retention schedule attached to it, specified by the -r flag. The retention schedule is used by the expiration process to, for example, expire all daily backups older than 10 days, or all monthly backups older than 6 months. Anything can be used for for a schedule name, but typical values are daily, weekly, monthly, yearly, and archive. If not specified, then the snebu-client script will automatically pick a retention schedule based on the date (Sunday through Friday are daily, Saturday is weekly, and the 1st of the month is monthly).

When a backup is requested, the backend will determine which files it already has (by comparing meta data such as file name, size, last modification time, and various other attributes). A list of files that are already on the backup server is added to the current backup set. The remaining files are then requested, and are added to the backup set when they are received.

File contents are stored and referenced by using the SHA1 hash of the file contents. Therefore, file level de-duplication is achieved across all backups stored on the backup server. In addition, files are stored compressed in an lzop compatible format. This allows for recovery of files even outside of the backup utility.

Setting up a local backup

This procedure assumes that both snebu and snebu-client are installed, and you will be backing up to a drive that is mounted under /media/snebu. These steps will be run as the root user. In the examples, the host name that is getting backed up is called zeus The backup retention schedules are automatically determined in these examples (use -r to override)

1. Create a user snebu, belonging to the group snebu.

2. Install the snebu package, or run make; make install from the source directory to install the system.

   This should install snebu in /usr/bin, or /usr/local/bin, with it owned by user:group snebu:snebu, and with file mode 4550.

3. Mount a drive under /media/snebu

4. Verify or create the configuration file /etc/snebu.conf with the following:

    meta=/media/snebu/catalog
    vault=/media/snebu/vault
   

   The meta directory is where the backup catalog is stored (in an SQLite DB). The vault directory contains all the backup file contents.

   Note: During operation, the backup catalog database receives a large number of random I/O operations. Therefore, if it is residing on a slower device, such as a 2.5" low-powered USB drive, the performance may be unacceptably slow. For this situation, better performance can be achieved by mounting an SSD on the catalog directory.

5. Create the directories from the snebu.conf file, and give ownership to the snebu user and group.

    mkdir -p /media/snebu/meta
    mkdir -p /media/snebu/vault
    chown -R snebu:snebu /media/snebu/
   

6. Create a file /etc/snebu-client.conf with the following: