STFS

Simple Tape File System (STFS), a file system for tapes and tar files.

Overview

STFS is a filesystem that brings tapes and tar files into the 21st century.

It enables you to:

• Use a tape or tar file like a regular disk: STFS uses the robust tar format and tape technology to provide a fully features filesystem. This makes such storage solutions much for accessible and manageable, while also significantly increasing the performance of everyday operations such as listing and searching for files by using a SQLite-based on-disk index.
• Archive data securely: The integrated transparent, asymmetrical encryption and signature support makes it possible to use almost any tape as a regulations compliant storage medium, while still being able to take advantage of all the benefits of tapes like reduced cost and increased reliability.
• Compress data efficiently: By leveraging the embedded compression functionality, it is possible to store even more data on tapes without sacrificing the user experience.
• Recover data in unexpected scenarios: Even if sudden power drops happen, the drive fails, the index gets corrupted or STFS stops being maintained, your data is secure. Because it is based on open standards such as tar, SQLite, zstandard, PGP and others, it is possible to extract your data even if STFS's integrated recovery tools don't suffice.
• Build your own storage solution: In addition to its own, optimized APIs, STFS provides a afero.FS implementation. This makes embedding STFS and accessing files on a tape or in a tar file through Go easy.

Installation

Static binaries are available on GitHub releases.

On Linux, you can install them like so:

$ curl -L -o /tmp/stfs "https://github.com/pojntfx/stfs/releases/latest/download/stfs.linux-$(uname -m)"
$ sudo install /tmp/stfs /usr/local/bin

On macOS, you can use the following:

$ curl -L -o /tmp/stfs "https://github.com/pojntfx/stfs/releases/latest/download/stfs.darwin-$(uname -m)"
$ sudo install /tmp/stfs /usr/local/bin

On Windows, the following should work (using PowerShell as administrator):

PS> Invoke-WebRequest https://github.com/pojntfx/stfs/releases/latest/download/stfs.windows-x86_64.exe -OutFile \Windows\System32\stfs.exe

Note that only the Linux version supports reading from tape drives; macOS and Windows are limited to operating on tar files.

You can find binaries for more operating systems and architectures on GitHub releases.

Tutorial

Please note that this is only a short overview and does not explain all configuration options. To get more info on available commands or options, use --help.

1. Generating Keys with stfs keygen

While not strictly required, it is recommended to generate keys to sign and encrypt your data on tape. There are multiple methods available; PGP and age for encryption, and PGP as well as Minisign for signatures. In most cases, using age for encryption and PGP for signatures is the best option. To generate the appropriate keys, run the following; make sure to save the keys in a secure location and use a secure password:

$ stfs keygen --encryption age --password mysecureencryptionpassword --identity ~/.stfs-age.priv --recipient ~/.stfs-age.pub
$ stfs keygen --signature pgp --password mysecuresignaturepassword --identity ~/.stfs-pgp.priv --recipient ~/.stfs-pgp.pub

For more information, see the key generation reference.

2. Serving a Tape Read-Write with stfs serve ftp

The simplest way to read or write to/from the tape (or tar file) is to use the integrated FTP server. To speed up operations, caching mechanisms and compression are available. For the write cache (--cache-write-type) the following types are available:

• memory: A simple in-memory cache; should not be used in most cases due to potential RAM exhaustion when adding large files
• file: A on-disk cache; this is recommended in most cases, especially if a SSD is available

For the read cache (--cache-filesystem-type), which is especially useful when working with many small files, similar types are available (memory and dir). dir uses a overlay filesystem to cache files in the directory specified with --cache-dir.

To further speed up IO-limited read/write operations, multiple compression options are available to be selected with --compression and can be tuned with --compression-level:

• zstandard: A Meta-led replacement for gzip with very high speeds and a very good compression ratio; this is recommended for most users
• gzip/parallelgzip: The GNU format commonly used in combination with tar, i.e. for .tar.gz; reasonably fast and with a good compression ratio
• bzip2/parallelbzip2: A reliable compression format with good speeds and a better compression ratio than gzip.
• lz4: Very fast, but at the cost of a lower compression ratio
• brotli: A Google-led compression format with good adoption on the web platform; very high compression ratio, very slow speeds

To serve a tape (or tar file), run the following (adjust the options accordingly):

# Use `-d /dev/nst0` for your primary tape drive instead
$ stfs serve ftp \
    -d ~/Downloads/drive.tar \
    -m ~/Downloads/metadata.sqlite \
    -e age \
    --encryption-identity ~/.stfs-age.priv \
    --encryption-recipient ~/.stfs-age.pub \
    --encryption-password mysecureencryptionpassword \
    -s pgp \
    --signature-identity ~/.stfs-pgp.priv \
    --signature-recipient ~/.stfs-pgp.pub \
    --signature-password mysecuresignaturepassword \
    --compression zstandard
{"time":1652646259,"level":"INFO","event":"FTP server listening","data":[{"laddr":":1337"}]}
{"time":1652646259,"level":"INFO","event":"Listening...","data":["address",{"IP":"::","Port":1337,"Zone":""}]}
{"time":1652646259,"level":"INFO","event":"Starting...","data":null}

You can now point your file manager (GNOME files on Linux, Windows Explorer on Windows and Finder on macOS all have support for it, but macOS is read-only) to ftp://localhost:1337 and read/write files from the tape (or tape file).

For more information, see the servers reference.

3. Serving a Tape Read-Only with stfs serve http

If you want to serve a tape (or tar file) read-only, using the integrated HTTP server is the best option. It inherits all the same options from Serving a Tape Read-Write with stfs serve ftp, minus the write cache due to it being read-only. To use it, run:

# Use `-d /dev/nst0` for your primary tape drive instead
$ stfs serve http \
    -d ~/Downloads/drive.tar \
    -m ~/Downloads/metadata.sqlite \
    -e age \
    --identity ~/.stfs-age.priv \
    --password mysecureencryptionpassword \
    -s pgp \
    --recipient ~/.stfs-pgp.pub \
    --compression zstandard
{"time":1652653259,"level":"INFO","event":"HTTP server listening","data":[{"laddr":":1337"}]}

You can now point your web browser to http://localhost:1337 and read files from the tape (or tape file).

For more information, see the servers reference.

4. Using Optimized Operations with stfs operation

While the file system API is convenient because of its similarity to most filesystems, it also can't be used without a write cache. While this isn't an issue for most applications, it requires you to have a disk that is at least as large as the largest file you want to add to the tape. To get around these limitations, STFS also provides a tar-like interface for interacting with the tape. Please note that these operations should be used carefully, as the usual checks (such as checking if a parent directory exists before adding files to it) don't apply.

First, initialize an empty tape:

# Use `-d /dev/nst0` for your primary tape drive instead
$ stfs operation initialize \
    -d ~/Downloads/drive.tar \
    -m ~/Downloads/metadata.sqlite \
    -e age \
    --recipient ~/.stfs-age.pub \
    -s pgp \
    --identity ~/.stfs-pgp.priv \
    --password mysecuresignaturepassword \
    --compression zstandard
type,indexed,record,lastknownrecord,block,lastknownblock,typeflag,name,linkname,size,mode,uid,gid,uname,gname,modtime,accesstime,changetime,devmajor,devminor,paxrecords,format
archive,false,-1,-1,-1,-1,53,/,,0,511,1000,1000,pojntfx,1000,2022-05-16T22:24:13+02:00,0001-01-01T00:00:00Z,0001-01-01T00:00:00Z,0,0,null,4
archive,true,0,-1,0,-1,53,/,,0,511,1000,1000,pojntfx,1000,2022-05-16T22:24:13+02:00,0001-01-01T00:00:00Z,0001-01-01T00:00:00Z,0,0,null,4

You can now add files to it:

# Use `-d /dev/nst0` for your primary tape drive instead
$ stfs operation archive \
    -d ~/Downloads/drive.tar \
    -m ~/Downloads/metadata.sqlite \
    -e age \
    --recipient ~/.stfs-age.pub \
    -s pgp \
    --identity ~/.stfs-pgp.priv \
    --password mysecuresignaturepassword \
    --compression zstandard \
    .
# ...
archive,true,1480,-1,9,-1,48,pkg/tape/write.go,,1544,420,1000,1000,pojntfx,pojntfx,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,0,0,"{""STFS.Signature"":""wnUEABYIACcFAmKCsyUJkGA0c/4XcV5qFqEEjWKRLHhppJ6S+ZJlYDRz/hdxXmoAAF5mAP95DKo/r136fL/SKuBwmxoMNfGZ+v61bwk/xcOBQk5vrwEAs07QV2RF6h/FME+/nXxjZrbBWmFWg8pC4IGdScnJbQ4="",""STF