HYPER-CUBE: High-Dimensional Hypervisor Fuzzing

<a href="https://www.ndss-symposium.org/wp-content/uploads/2020/02/23096-paper.pdf"> <img align="right" width="200" src="paper.png"> </a>

Hyper-Cube is a black-box fuzzer designed specifically to test x86 hypervisors. Our approach is based on a custom operating system that implements a custom bytecode interpreter to enable fuzzing of emulated hypervisor components. The OS is tailored to work with most x86 hypervisors that support either BIOS or 32-bit UEFI boot. Further, it implements basic features such as ACPI and PCI device enumeration to automatically detect potential MMIO (memory-mapped I/O) and PIO (port I/O) areas for later fuzzing. As a result, Hyper-Cube has helped to uncover crashes and vulnerabilities in different hypervisors such as QEMU, bhyve, Virtual Box, VMWare, ACRN, and Parallels.

Please note that this version of Hyper-Cube is only designed for black-box fuzzing. Support for feedback fuzzing has later been added in our follow-up paper Nyx. In that paper, we covered how we implemented feedback fuzzing support for type-2 hypervisors by employing different techniques, such as fast VM snapshots and Intel PT support. Like Hyper-Cube, the fuzzing is also performed by a custom OS, which is based on a slightly modified version of Hyper-Cube. The source code of this modified version can be found here. However, unlike Nyx, Hyper-Cube can be used with most hypervisors without much hassle, simply by booting it in the target hypervisor.

@inproceedings{hypercube,
  title={HYPER-CUBE: High-Dimensional Hypervisor Fuzzing.},
  author={Schumilo, Sergej and Aschermann, Cornelius and Abbasi, Ali and W{\"o}rner, Simon and Holz, Thorsten},
  booktitle={NDSS},
  year={2020}
  url = {https://www.ndss-symposium.org/ndss-paper/hyper-cube-high-dimensional-hypervisor-fuzzing/},
}

Setup:

Run the provided setup.sh script to install all required dependencies. This script will take care of all dependencies, prepare and compile GRUB, and set up everything else required. Please note that this script is specifically designed to work with Ubuntu 16.04 or 18.04 (similar to Hyper-Cube, which has only been tested with these two Ubuntu versions).

cd hypercube/
./setup.sh

Getting Started:

In the following section, we will outline how to use the fuzzing capabilities of Hyper-Cube. It is important to note that certain features, such as input minimization, currently only work with QEMU (more on that later).

In general, the structure of this project can be divided into two main components: first, there is the main component of the project, which consistes of the actual operating system and the fuzzer (both are mainly written in C). The code can be found here. The second part consists of additional tooling written in Python, which can be accessed here. The Python tool serves the purpose of automating specific tasks, such as building the OS image, and in addition, it also implements high-level features, such as the input minimizer.

Please note that some features currently only work with QEMU. This limitation arises from the necessity of a target-specific runner module, which must be implemented in Python. The purpose of a target runner is to launch the target hypervisor and provide information in the event of crashes. Additionally, it also implements features such as timeout detection and so on. The QEMU target runner implements all of that using ptrace. The code of the QEMU runner can be found here. Besides, a custom target runner is not required to run the fuzzer.

General Usage:

Hyper-Cube can be easily configured to target a single or even multiple different PIO and MMIO areas at the same time. For this purpose, we utilize the Python tooling to generate a custom build of Hyper-Cube OS specifically tailored for our following use case.

While it is also possible to perform the build procedure manually by compiling and packaging the code located in the ./os/ directory by yourself; however, we strongly recommend utilizing the Python tool instead. Manually configuring Hyper-Cube OS also requires a C-header file that acts as a configuration and is typically automatically generated by the Python tooling.

In the following example, we will generate an OS image that targets the VGA device emulator. For this purpose, we configure Hyper-Cube to automatically initiate PCI device enumeration, detecting all available PCI devices once the boot process is complete. Additionally, we apply a filter set to VGA to limit the fuzzer to interact only with PCI devices related to the emulated graphics card.

python3 hypercube.py generate /tmp/vga.iso --enable_pci --filter "VGA"

This OS image can now be used with any hypervisor or emulator that supports x86 BIOS boot. Once the OS image is booted, you will see various visual VGA artifacts, due to the fuzzer performing random operations on the associated PIO and MMIO regions. In case a bug or vulnerability exists in the emulator implementation, this can also lead to a crash (such as CVE-2018-3223). If the target hypervisor supports EFI boot for 32-bit operating systems, you can generate an EFI-compatible image by enabling the --uefi option. Please note that 64-bit EFI boot is not supported.

Moreover, you can also enable the enumeration of other device types, such as legacy I/O or x86 platform-specific devices like HPET or APIC. The enumeration of these device types can be enabled by setting the corresponding options via command line arguments. To get a comprehensive list of all available options, run the following command:

python3 hypercube.py generate --help

This command will result in the following output:

...
  --enable_io           enable IO enumeration
  --enable_default_io   enable default IO enumeration
  --enable_pci          enable PCI enumeration
  --enable_e_pci        enable extended PCI
  --enable_apic         enable APIC
  --enable_hpet         enable HPET
...

Additionally, if needed, one can manually configure target areas as follows:

  --extra_areas <base> <length> <type> <name>
    # base:   physical address of the target area (hexadecimal)
    # lenght: size of the target area (hexadecimal)
    # type:   0=PIO / 1=MMIO
    # name:   name of this specific area

Moreover, to output a comprehensive list of all detected and configured PIO and MMIO areas, it is recommended to create an image without enabling the actual fuzzer. This is especially useful for VGA or serial device fuzzing, as it allows to manually check the configuration once the OS is booted. This can be done by setting the --disable_fuzzer flag. Upon booting the OS, the information on detected and configured areas will be reported through either the serial interface or VGA (if the --enable_vga option is enabled).

Furthermore, after the OS has been booted, it will print the seed value of the RNG used by the fuzzer. This seed value can later be used to reproduce specific crashes. For instance, you can generate a new OS image with a fixed seed value by setting the --seed <value> argument. This seed value is also used to run the input minimization, aiming to find a minimal crash reproducer.

Target QEMU's megasas Emulator:

In the following section, we will describe various use cases by targeting QEMU's implementation of the megasas RAID emulator. Additionally, we will demonstrate how to use the test case minimizer with the bugs found.

Run the Fuzzer:

We start with fuzzing the megasas device emulator in QEMU 4.0.0. To do this, we first need to compile an ASAN build of QEMU 4.0.0 as our target hypervisor for this example. Please note that compiling QEMU 4.0.0 is not supported on Ubuntu 16.04.

cd targets/
./compile_qemu.sh 4.0.0 asan
cd -

Next, we generate a new OS image with enabled PCI device enumeration and a set filter for PCI class names containing "RAID":

python3 hypercube.py generate /tmp/megasas.iso --enable_pci --filter "RAID"

Now we boot the OS and launch the fuzzer with the following command.

python3 hypercube.py run qemu-4.0.0-asan /tmp/megasas.iso --qemu_params " -device megasas"

This command will work only with QEMU on Linux because it requires a target runner, that is currently only implemented for QEMU. As an alternative, you can manually launch QEMU with Hyper-Cube OS by using the following command:

targets/qemu-4.0.0-asan/x86_64-softmmu/qemu-system-x86_64 -cdrom /tmp/megasas.iso -serial mon:stdio -m 100 -nographic -net none  -device megasas

Input Minimization & Testcase Decompiler:

After some time, with the command above, the fuzzer will trigger a crash in QEMU. A crash can be reproduced by using the same seed value of a given run (in our example, we pick 0x680aaa46, which results in a heap use-after-free). We can generate a new image with a fixed seed value, using the following command:

python3 hypercube.py generate /tmp/test.iso --enable_pci --filter "RAID" --seed 0x680aaa46

Moreover, the input minimizer can also be utilized to produce a minimal test case reproducer for this specific crash. Similar to the previous command, the test case reproducer also expects an RNG seed.

Hyper-Cube OS uses a bytecode interpreter called Tesseract to convert a stream of random data into a sequence of fuzzing OP codes (more details on that are described in our paper). This stream is basically the output of a RNG seeded by either a random or fixed value. Instead of a data stream, the input minimizer will try to find a fragment from that stream reproducing the same crash. So, instead of using a fixed seed to reproduce a crash, a test case file can also be embedded in the OS image to get the