Convenience of containers, security of virtual machines

With firebuild, you can build and deploy secure VMs directly from Dockerfiles and Docker images in just few minutes.

The concept of firebuild is to leverage as much of the existing Docker world as possible. There are thousands of Docker images out there. Docker images are awesome because they encapsulate the software we want to run in our workloads, they also encapsulate dependencies. Dockerfiles are what Docker images are built from. Dockeriles are the blueprints of the modern infrastructure. There are thousands of them for almost anything one can imagine and new ones are very easy to write.

With firebuild it is possible to:

• build root file systems directly from Dockerfiles
• tag and version root file systems
• run and manage microvms on a single host
• define run profiles

High level example

Build and start HashiCorp Consul 1.9.4 on Firecracker with three simple steps:

• build a base operating system image
• build Consul image
• start the application

sudo $GOPATH/bin/firebuild baseos \
    --profile=standard \
    --dockerfile $(pwd)/baseos/_/alpine/3.12/Dockerfile

sudo $GOPATH/bin/firebuild rootfs \
    --profile=standard \
    --dockerfile=git+https://github.com/hashicorp/docker-consul.git:/0.X/Dockerfile \
    --cni-network-name=machine-builds \
    --ssh-user=alpine \
    --vmlinux-id=vmlinux-v5.8 \
    --tag=combust-labs/consul:1.9.4

sudo $GOPATH/bin/firebuild run \
    --profile=standard \
    --name=consul1 \
    --from=combust-labs/consul:1.9.4 \
    --cni-network-name=machines \
    --vmlinux-id=vmlinux-v5.8

Find the IP of the consul1 VM and query Consul:

VMIP=$(sudo $GOPATH/bin/firebuild inspect \
    --profile=standard \
    --vmm-id=consul1 | jq '.NetworkInterfaces[0].StaticConfiguration.IPConfiguration.IP' -r)

$ curl http://${VMIP}:8500/v1/status/leader
"127.0.0.1:8300"

But how?

clone and build from sources

mkdir -p $GOPATH/src/github.com/combust-labs/firebuild
cd $GOPATH/src/github.com/combust-labs/firebuild
go install

The binary will be placed in $GOPATH/bin/firebuild.

create a profile

# create required directories, these need to exist before the profile can be created:
sudo mkdir -p /firecracker/rootfs
sudo mkdir -p /firecracker/vmlinux
sudo mkdir -p /srv/jailer
sudo mkdir -p /var/lib/firebuild
# create a profile:
sudo $GOPATH/bin/firebuild profile-create \
	--profile=standard \
	--binary-firecracker=$(readlink /usr/bin/firecracker) \
	--binary-jailer=$(readlink /usr/bin/jailer) \
	--chroot-base=/srv/jailer \
	--run-cache=/var/lib/firebuild \
	--storage-provider=directory \
	--storage-provider-property-string="rootfs-storage-root=/firecracker/rootfs" \
	--storage-provider-property-string="kernel-storage-root=/firecracker/vmlinux" \
	--tracing-enable

Kernel images will be stored in /firecracker/vmlinux, root file systems will be stored in /firecracker/rootfs.

build the kernel

The examples use the 5.8 Linux kernel image which is built using the configuration from the baseos/kernel/5.8.config file in this repository. To build the kernel:

export KERNEL_VERSION=v5.8
mkdir -p /tmp/linux && cd /tmp/linux
git clone https://github.com/torvalds/linux.git .
git checkout ${KERNEL_VERSION}
wget -O .config https://raw.githubusercontent.com/combust-labs/firebuild/master/baseos/kernel/5.8.config
make vmlinux -j32 # adapt to the number of cores you have

Once built, copy the kernel to the storage:

mv /tmp/linux/vmlinux /firecracker/vmlinux/vmlinux-${KERNEL_VERSION}

setup CNI

firebuild assumes CNI availability. Installing the plugins is very straightforward. Create /opt/cni/bin/ directory and download the plugins:

mkdir -p /opt/cni/bin
curl -O -L https://github.com/containernetworking/plugins/releases/download/v0.9.1/cni-plugins-linux-amd64-v0.9.1.tgz
tar -C /opt/cni/bin -xzf cni-plugins-linux-amd64-v0.9.1.tgz

Firecracker also requires the tc-redirect-tap plugin. Unfortunately, this one does not offer downloadable binaries and has to be built from sources.

mkdir -p $GOPATH/src/github.com/awslabs/tc-redirect-tap
cd $GOPATH/src/github.com/awslabs/tc-redirect-tap
git clone https://github.com/awslabs/tc-redirect-tap.git .
make install

create a dedicated CNI network for the builds

Feel free to change the ipam.subnet or set multiple ones. host-local IPAM CNI plugin documentation.

cat <<EOF > /etc/cni/conf.d/machine-builds.conflist
{
    "name": "machine-builds",
    "cniVersion": "0.4.0",
    "plugins": [
        {
            "type": "bridge",
            "name": "builds-bridge",
            "bridge": "builds0",
            "isDefaultGateway": true,
            "ipMasq": true,
            "hairpinMode": true,
            "ipam": {
                "type": "host-local",
                "subnet": "192.168.128.0/24",
                "resolvConf": "/etc/resolv.conf"
            }
        },
        {
            "type": "firewall"
        },
        {
            "type": "tc-redirect-tap"
        }
    ]
}
EOF

caution

The maximum socket path in the Linux Kernel is 107 characters + \0:

struct sockaddr_un {
	__kernel_sa_family_t sun_family; /* AF_UNIX */
	char sun_path[UNIX_PATH_MAX];	/* pathname */
};

The --chroot-base value must have a maximum length of 31 characters. The constant jailer path suffix used by firebuild is 76 characters:

• constant /firecracker-v0.22.4-x86_64/ (automatically generated by the jailer)
• VM ID is always 20 characters long
• constant /root/run/firecracker.socket assumed by the jailer

Example: /firecracker-v0.22.4-x86_64/sifuqm4rq2runxparjcx/root/run/firecracker.socket.

Using more than 31 characters for the --chroot-base value, regardless if in the profile setting or using the command --chroot-base flag, will lead to a very obscure error. Firecracker will report an error similar to:

INFO[0006] Called startVMM(), setting up a VMM on /mnt/sdd1/firebuild/jailer/firecracker-v0.22.4-x86_64/6b41ecc3783c4f38a743c9c8af4bbe0f/root/run/firecracker.socket
WARN[0009] Failed handler "fcinit.StartVMM": Firecracker did not create API socket /mnt/sdd1/firebuild/jailer/firecracker-v0.22.4-x86_64/6b41ecc3783c4f38a743c9c8af4bbe0f/root/run/firecracker.socket: context deadline exceeded
{"@level":"error","@message":"Firecracker VMM did not start, build failed","@module":"rootfs","@timestamp":"2021-03-14T19:20:49.856228Z","reason":"Failed to start machine: Firecracker did not create API socket /mnt/sdd1/firebuild/jailer/firecracker-v0.22.4-x86_64/6b41ecc3783c4f38a743c9c8af4bbe0f/root/run/firecracker.socket: context deadline exceeded","veth-name":"vethHvfZiskhLkQ","vmm-id":"6b41ecc3783c4f38a743c9c8af4bbe0f"}
{"@level":"info","@message":"cleaning up jail directory","@module":"rootfs","@timestamp":"2021-03-14T19:20:49.856407Z","veth-name":"vethHvfZiskhLkQ","vmm-id":"6b41ecc3783c4f38a743c9c8af4bbe0f"}
{"@level":"info","@message":"cleaning up temp build directory","@module":"rootfs","@timestamp":"2021-03-14T19:20:49.856458Z"}
WARN[0010] firecracker exited: signal: killed

In the above example, the path is 114 characters long. Changing the chroot to /mnt/sdd1/fc/jail would solve the problem.

build the base operating system root file system

firebuild uses the Docker metaphor. An image of an application is built FROM a base. An application image can be built FROM alpine:3.13, for example. Or FROM debian:buster-slim, or FROM registry.access.redhat.com/ubi8/ubi-minimal:8.3 and dozens others.

In order to fulfill those semantics, a base operating system image must be built before the application root file system can be created.

Build a base Debian Buster slim:

sudo $GOPATH/bin/firebuild baseos \
    --profile=standard \
    --dockerfile $(pwd)/baseos/_/debian/buster-slim/Dockerfile

Because the baseos root file system is built completely with Docker, there is no need to configure the kernel storage.

This does not belong here, structure better: It's possible to tag the baseos output using the --tag= argument, for example:

sudo $GOPATH/bin/firebuild baseos \
    --profile=standard \
    --dockerfile $(pwd)/baseos/_/debian/buster-slim/Dockerfile \
    --tag=custom/os:latest

create a Postgres 13 VM rootfs directly from the upstream Dockerfile

The upstream Dockerfile is built FROM debian:buster-slim, that's the baseos built in the previous step:

sudo $GOPATH/bin/firebuild rootfs \
    --profile=standard \
    --dockerfile=git+https://github.com/docker-library/postgres.git:/13/Dockerfile \
    --cni-network-name=machine-builds \
    --vmlinux-id=vmlinux-v5.8 \
    --mem=512 \
    --tag=combust-labs/postgres:13

create a separate CNI network for running VMs

For example:

cat <<EOF > /etc/cni/conf.d/machines.conflist
{
    "name": "machines",
    "cniVersion": "0.4.0",
    "plugins": [
        {
            "type": "bridge",
            "name": "machines-bridge",
            "bridge": "machines0",
            "isDefaultGateway": true,
            "ipMasq": true,
            "hairpinMode": true,
            "ipam": {
                "type": "host-local",
                "subnet": "192.168.127.0/24",
                "resolvConf": "/etc/resolv.conf"
            }
        },
        {
            "type": "firewall"
        },
        {
            "type": "tc-redirect-tap"
        }
    ]
}
EOF

run the VM from the resulting tag

Once the root file system is built, start the VM:

sudo $GOPATH/bin/firebuild run \
    --profile=standard \
    --name=postgres1 \
    --from=combust-labs/postgres:13 \
    --cni-network-name=machines \
    --vmlinux-id=vmlinux-v5.8 \
    --mem=512 \
    --env="POSTGRES_PASSWORD