<img width="100" align="right" src="img/hermitcore_logo.png" />

DEPRECATION NOTICE:

We have shifted our efforts from this project to the rusty-hermit project, which is also a unikernel but rewritten in Rust. This project will not receive any active contributions from the maintainers and only limited responses/help in GitHub issues.

HermitCore - A lightweight unikernel for a scalable and predictable runtime behavior

The project HermitCore is a new unikernel targeting a scalable and predictable runtime for high-performance and cloud computing. HermitCore extends the multi-kernel approach (like McKernel) with unikernel features for a better programmability and scalability for hierarchical systems.

On the startup of HermitCore applications, cores are isolated from the Linux system enabling bare-metal execution of on these cores. This approach achieves lower OS jitter and a better scalability compared to full-weight kernels. Inter-kernel communication between HermitCore applications and the Linux system is realized by means of an IP interface.

In addition to the multi-kernel approach described above, HermitCore can be used as a classical standalone unikernel as well. In this case, HermitCore runs a single-kernel exclusively on the hardware or within a virtual machine. This reduces the resource demand and lowers the boot time which is critical for cloud computing applications. It is the result of a research project at RWTH Aachen University and is currently an experimental approach, i.e., not production ready. Please use it with caution.

Contributing

HermitCore is being developed on GitHub. Create your own fork, send us a pull request, and chat with us on Slack.

Requirements

The build process works currently only on x86-based Linux systems. To build the HermitCore kernel and applications you need:

• CMake
• Netwide Assember (NASM)
• recent host compiler such as GCC
• HermitCore cross-toolchain, i.e. Binutils, GCC, newlib, pthreads
• Python3

HermitCore cross-toolchain

We provide prebuilt packages for Ubuntu 18.04 and Debian 9 of the HermitCore toolchain, which can be installed as follows:

Ubuntu 18.04

$ echo "deb [trusted=yes] https://dl.bintray.com/hermitcore/ubuntu bionic main" | sudo tee -a /etc/apt/sources.list
$ sudo apt-get -qq update
$ sudo apt-get install binutils-hermit newlib-hermit pte-hermit gcc-hermit libomp-hermit libhermit

Debian 9

$ echo "deb [trusted=yes] https://dl.bintray.com/hermitcore/debian stretch main" | sudo tee -a /etc/apt/sources.list
$ sudo apt-get -qq update
$ sudo apt-get install binutils-hermit newlib-hermit pte-hermit gcc-hermit libomp-hermit libhermit

For unsupported systems, a docker image with the complete toolchain is provided and can be installed as follows:

$ docker pull rwthos/hermitcore

The following commad starts within the new docker container a shell and mounts from the host system the directory ~/src to /src:

$ docker run -i -t -v ~/src:/src rwthos/hermitcore:latest

Within the shell the cross-toolchain can be used to build HermitCore applications.

If you want to build the toolchain yourself, have a look at the repository hermit-toolchain, which contains scripts to build the whole toolchain.

Depending on how you want to use HermitCore, you might need additional packages such as:

• QEMU (apt-get install qemu-system-x86)

Building HermitCore

Preliminary work

To build HermitCore from source (without compiler), the repository with its submodules has to be cloned.

$ git clone git@github.com:hermitcore/libhermit.git HermitCore
$ cd HermitCore
$ git submodule init
$ git submodule update

We require a fairly recent version of CMake (3.7) which is not yet present in most Linux distributions. We therefore provide a helper script that fetches the required CMake binaries from the upstream project and stores them locally, so you only need to download it once.

$ . cmake/local-cmake.sh
-- Downloading CMake
--2017-03-28 16:13:37--  https://cmake.org/files/v3.7/cmake-3.7.2-Linux-x86_64.tar.gz
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving cmake.org... 66.194.253.19
Connecting to cmake.org|66.194.253.19|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30681434 (29M) [application/x-gzip]
Saving to: ‘cmake-3.7.2-Linux-x86_64.tar.gz’

cmake-3.7.2-Linux-x86_64.tar.gz         100%[===================>]  29,26M  3,74MB/s    in 12s

2017-03-28 16:13:50 (2,48 MB/s) - ‘cmake-3.7.2-Linux-x86_64.tar.gz’ saved [30681434/30681434]

-- Unpacking CMake
-- Local CMake v3.7.2 installed to cmake/cmake-3.7.2-Linux-x86_64
-- Next time you source this script, no download will be necessary

So before you build HermitCore you have to source the local-cmake.sh script everytime you open a new terminal.

Building the library operating systems and its examples

To build HermitCore go to the directory with the source code, create a build directory, and call in the new directory cmake followed by make.

$ mkdir build
$ cd build
$ cmake ..
$ make
$ sudo make install

If your toolchain is not located in /opt/hermit/bin then you have to supply its location to the cmake command above like so:

$ cmake -DTOOLCHAIN_BIN_DIR=/home/user/hermit/bin ..

Assuming that binaries like x86_64-hermit-gcc and friends are located in that directory. To install your new version in the same directory, you have to set the installation path and to install HermitCore as follows:

$ cmake -DTOOLCHAIN_BIN_DIR=/home/user/hermit/bin -DCMAKE_INSTALL_PREFIX=/home/user/hermit ..
$ make
$ make install

Note: If you use the cross compiler outside of this repository, the compiler uses per default the library operating systems located by the toolchain (e.g. /opt/hermit/x86_64-hermit/lib/libhermit.a).

Starting Applications with the Proxy tool

HermitCore applications are currently started with a small helper tool called proxy. This tool sets up a virtual machine or bare-metal on a NUMA node and bridges certain functionality from the application to the Linux host.

To start a HermitCore application, hand the executable to the proxy:

/opt/hermit/bin/proxy myHermiCoreApplication

Optional: The proxy can be registered as loader for HermitCore elf files to the Linux system via binfmt_misc, so that HermitCore applications can be started like common Linux applications.

$ sudo -c sh 'echo ":hermit:M:7:\\xff::/opt/hermit/bin/proxy:" > /proc/sys/fs/binfmt_misc/register'
$ # dirct call of a HermitCore application
$ /opt/hermit/x86_64-hermit/extra/tests/hello
Hello World!!!

The proxy starts the HermitCore applications within a VM. The application host (QEMU, Uhyve or bare-metal on a NUMA node as multi-kernel) can be set via the environment variable $HERMIT_ISLE.

$ # using QEMU
$ HERMIT_ISLE=qemu /opt/hermit/bin/proxy /opt/hermit/x86_64-hermit/extra/tests/hello

Testing

As classical standalone unikernel within a virtual machine

HermitCore applications can be directly started as standalone kernel within a virtual machine. In this case, iRCCE is not supported.

$ cd build
$ make install DESTDIR=~/hermit-build
$ cd ~/hermit-build/opt/hermit
$ # using QEMU
$ HERMIT_ISLE=qemu bin/proxy x86_64-hermit/extra/tests/hello
$ # using uHyve
$ HERMIT_ISLE=uhyve bin/proxy x86_64-hermit/extra/tests/hello

With HERMIT_ISLE=qemu, the application will be started within a QEMU VM. Please note that the loader requires QEMU and uses per default KVM. Furthermore, it expects that the executable is called qemu-system-x86_64.

With HERMIT_ISLE=uhyve, the application will be started within a thin hypervisor powered by Linux's KVM API and therefore requires KVM support. uhyve has a considerably smaller startup time than QEMU, but lacks some features such as GDB debugging. In principle, it is an extension of ukvm.

In this context, the environment variable HERMIT_CPUS specifies the number of cpus (and no longer a range of core ids). Furthermore, the variable HERMIT_MEM defines the memory size of the virtual machine. The suffix of M or G can be used to specify a value in megabytes or gigabytes respectively. Per default, the loader initializes a system with one core and 2 GiB RAM. For instance, the following command starts the stream benchmark in a virtual machine, which has 4 cores and 6GB memory.

$ HERMIT_ISLE=qemu HERMIT_CPUS=4 HERMIT_MEM=6G bin/proxy x86_64-hermit/extra/benchmarks/stream

To enable an ethernet device for uhyve, we have to setup a tap device on the host system. For instance, the following command establish the tap device tap100 on Linux:

$ sudo ip tuntap add tap100 mode tap
$ sudo ip addr add 10.0.5.1/24 broadcast 10.0.5.255 dev tap100
$ sudo ip link set dev tap100 up
$ sudo bash -c 'echo 1 > /proc/sys/net/ipv4/conf/tap100/proxy_arp'

Per default, uhyve's network interface uses 10.0.5.2as IP address, 10.0.5.1 for the gateway and 255.255.255.0 as network mask. The default configuration could be overloaded by the environment va