usbd

A User-Space Block Device (USBD) Framework (written in Go)

Introduction

Many people are familiar with user-space filesystem; on Linux these are popularly implemented using FUSE. This library is an attempt to similar provide similar facility for user-space block devices, specifically those written in Go. USBD takes advantage of the seldom used NBD interface provided by an in tree Linux kernel module to allow a daemon running in user-space to export a block device. This package can be used to write software block device (in Go) and export it via /dev/nbdX where X is the next available device on the system. After doing so, such a device can be formatted with the filesystem of the user's choice (ex. ext4, btrfs, xfs, zfs, etc) and mounted in the usual ways (ex. mount, /etc/fstab, etc).

A daemon is created using the USBD framework that acts as an NBD server. Overhead between the user-space daemon and the kernel is minimized by using the AF_UNIX (also known as AF_LOCAL) socket type to communicate between local processes efficiently. The USBD framework allows a Go programmer to define their own block devices by implementing a type that conforms the following simple interface:

type Device interface {
	Size() int64
	BlockSize() int64
	ReadAt(buf []byte, pos int64) (count int, err error)
	WriteAt(buf []byte, pos int64) (count int, err error)
	Trim(pos int64, count int) error
	Flush() error
	Close() error
}

Typically a daemon will call NewNbdHandler to instantiate a new usbdlib.NbdStream which is configured to communicate with the kernel NDB module and is passed to usbdlib.ReqProcessor by calling NbdStream.ProcessRequests which acts as a proxy between the kernel which is handling NDB requests and the users usbdlib.Device implementation. A command-line server, usbdsrvd, is included to allow easy testing of the USBD engine and reference devices. In its implementation the above initialization procedure can be observed. This server will make a chosen device available as /dev/nbdX after which time it can be formatted and mounted or otherwise used as any other block device.

Project Structure

cmd
├── usbdsrvd
│   │   └── An example daemon that use usbdlib to serve this repositories user-space
│   │       device implementations.
│   └── conf
│       └── usbdsrvd configuration definitions and logic.
pkg
├── devices
│   ├── filedisk
│   │   └── An example USBD implementation that is backed by a simple file (filesystem).
│   ├── dedupdisk
│   │   └── An example, but potentially useful, USBD implementation that uses files to
│   │       back a logical device, but with the added use of hashing and a Pebble database
│   │       for block level deduplication. This is an older prototype and file/ramdisk are
│   │       better examples for those looking to implement their own devices.
│   ├── ramdisk
│   │   └── An example USBD implementation that is memory backed (ramdrive).
│   └── objstore
│       └── A USBD implementation, which is practically useful as it creates a logical
│           device which is cached locally but actually backed by a remote objectstore
│           of the users choice. (Ex. S3/minio, Swift, BackBlaze, Azure/Google Object Storage).
│           However, due to its complexity is not a great example for learning how to build
│           a user-space block device.
├── usbdlib
│   └── The core USBD interface definitions and engine to export/serve implementations
└── util
    └── Utilities for implementors of user-space block devices to reuse    

Note the objstore and dedupdisk implementations are non-trivial implementation compared to the others and use the stow ObjectStorage abstraction library and Pebble DB respectively.

Current state

As of writing, this package is actively maintained. Issue reports are welcome. Pull requests are welcome but will require consent to a contributor agreement. This project is currently beta quality. There are no known defects or risks to usage, but it has not yet been widely used. Experience reports are helpful and encouraged!

Basic Reference Devices (ramdisk and filedisk)

There are ramdisk and filedisk device implementations that are excellent for understanding how to implement a simple USBD device. Starting usbdsrvd with defaults (no arguments) will result in a 1 GB ramdisk backed device being exposed as the next available NBD device, typically /dev/nbd0. Unlike these straight forward examples, there are also some advanced device implemenations which may be of more practical value.

Building

This project is completely Linux centric (since it uses NBD) and requires the NBD kernel headers to be installed and available. To build the Go toolchain must be installed after that building is a simple matter of running go build in the usbdsrvd directory to generate an executable.

Testing

usbd has both automated unit testing and manual testing approaches.

Unit Testing

There is a provided a suite of basic unit tests for implementors of devices that all of the included devices use:

$ cd usbd
$ go test ./...
ok  	github.com/tarndt/usbd/pkg/devices/dedupdisk/dedupdisk_test	6.166s
ok  	github.com/tarndt/usbd/pkg/devices/filedisk/filedisk_test	8.011s
ok  	github.com/tarndt/usbd/pkg/devices/objstore	65.617s
ok  	github.com/tarndt/usbd/pkg/devices/ramdisk	2.351s
ok  	github.com/tarndt/usbd/pkg/usbdlib	(cached)
ok  	github.com/tarndt/usbd/pkg/usbdlib/usbdlib_test	0.197s
ok  	github.com/tarndt/usbd/pkg/util	(cached)

Executing these tests is just a matter of running go test. As seen above you can run all unit tests in the project repository by running go test ./... in the root directory. Some tests that interact with the Linux kernel require super-user privileges (aka root). Should you run tests with verbosity (go test -v), you will see some tests are skipped during normal execution. Additionally, while running tests with the race detector (-race) has proven fruitful for discovering data races, please be warned that in this mode tests often take an order of magnitude longer to run and may require you to increase the test timeout (ex. -timeout=10m). The provided test suits do support a short mode as well (go test -short). Sometimes its useful to compile a package's unit tests to its own executable, and this can be done in typical Go fashion go test -c (or go test -c -race to also enable the data race detector).

Manual Testing

Manual testing has been performed using standard block device tools (ex. hdparm) and by exporting an instance of the USBD reference implementations and creating a VirtualBox VM using the exported NDB device and installing Windows XP/10 and Ubuntu on it.

Advanced Devices

In addition to the basic reference devices, as mentioned in the pr