LLVM Multicompiler

This repo is based off the official LLVM git mirror: http://llvm.org/git/llvm.git. We have added passes which randomize the implementation details of the code to combat code-reuse attacks.

Test Environment

Ubuntu 14.04 LTS 64 bit

gcc 4.8.4

Python 2.7.6

GNU M4 1.4.17

GNU Autoconf 2.69

GNU Automake 1.14.1

libtool 2.4.2

zlib 1.2.3.4

binutils 2.24 (build instructions below)

Installation

Below are all instructions explained above.

Installing prerequisites

apt-get install -y libssl-dev libxml2-dev libpcre3-dev

Checking out LLVM, Clang, compiler-rt, poolalloc, and SVF:

git clone git@github.com:securesystemslab/multicompiler.git llvm

git clone git@github.com:securesystemslab/multicompiler-clang.git llvm/tools/clang

git clone git@github.com:securesystemslab/multicompiler-compiler-rt.git llvm/projects/compiler-rt

git clone git@github.com:securesystemslab/poolalloc.git llvm/projects/poolalloc

Link-Time Optimization (LTO)

Function randomization requires link-time optimization, since LTO ensures that functions are shuffled not within each source file but across the whole program, achieving higher entropy.

Installing prerequisites

apt-get install -y flex bison texinfo

Building binutils with Gold for LLVM LTO support and global shuffling

For Binutils 2.26, clone:

git clone git@github.com:/securesystemslab/binutils.git

Binutils 2.24 is also available in the cfar-2_24 branch

If you want to use Readactor execute-only features, apply the Readactor Gold patch now:

patch -p1 < PATH_TO_READACTOR_SOURCES/linker/binutils-gold-xonly.patch

Configure binutils with these flags:

--enable-gold --enable-plugins --prefix=prefix --disable-werror

(prefix is not required, you can run gold/ld-new directly from the build directory by symlinking or copying it to the LLVM build/install binary directory.)

The randomizing gold patch requires openssl < v1.1. To explicitly build against openssl-1.0, add the following environment variables before both the configure and make commands: CPPFLAGS="-I/usr/include/openssl-1.0" LDFLAGS="-L/usr/lib/openssl-1.0".

then

make

make install

Building and Compiling LLVM

Patch printf.h from glibc:

There is a bug in glibc printf.h: https://sourceware.org/bugzilla/show_bug.cgi?id=18907. Apply the following patch to /usr/include/printf.h to fix the bug.

--- /usr/include/printf.h.orig	2016-12-13 21:34:35.897301441 +0000
+++ /usr/include/printf.h	2016-12-13 21:35:40.374031243 +0000
@@ -111,13 +111,13 @@
    it returns a positive value representing the bit set in the USER
    field in 'struct printf_info'.  */

-extern int register_printf_modifier (const wchar_t *__str) __wur __THROW;
+extern int register_printf_modifier (const wchar_t *__str) __THROW __wur;


 /* Register variable argument handler for user type.  The return value
    is to be used in ARGINFO functions to signal the use of the
    type.  */
-extern int register_printf_type (printf_va_arg_function __fct) __wur __THROW;
+extern int register_printf_type (printf_va_arg_function __fct) __THROW __wur;


 /* Parse FMT, and fill in N elements of ARGTYPES with the

Recommended: Building LLVM and Clang using cmake:

Linux

1. cmake .. -DLLVM_TARGETS_TO_BUILD="X86" -DCMAKE_INSTALL_PREFIX=... -DCMAKE_BUILD_TYPE=Release -DLLVM_BINUTILS_INCDIR=...

To build cross-checking support, add the following to the previous command -DMULTICOMPILER_RAVEN_SRCDIR=<raven sources> -DMULTICOMPILER_RAVEN_OUTDIR=<raven build> where <raven sources> is the directory containing the raven MVEE source code (containing rbuff/rbuff.h) and <raven build> is the directory containing the built raven MVEE (containing rbuff/librbuff.so).

The multicompiler RNG requires openssl < v1.1. If you have both openssl-1.0 and openssl-1.1 installed, you need to force cmake to recognize the 1.0 version with the following additional arguments: -DOPENSSL_INCLUDE_DIR=/usr/include/openssl-1.0 -DOPENSSL_CRYPTO_LIBRARY=/usr/lib/libcrypto.so.1.0.0. If you wish to enable LLVM_OPTIMIZED_TABLEGEN, you'll also need to patch NATIVE/CMakeCache.txt in the build directory manually after the build fails because there is no good way to set cmake variables for the second cmake invocation the LLVM build system uses to build optimized tablegen for a debug build. Manually overwrite OPENSSL_CRYPTO_LIBRARY and OPENSSL_INCLUDE_DIR with the values from CMakeCache.txt. This has been tested on current Arch Linux, exact paths may vary for other distributions.

2. make

macOS

To build on macOS, you may need to export OPENSSL_ROOT_DIR so it points to your openssl root folder. Homebrew puts openssl in /usr/local/opt/openssl.

1. Export OPENSSL_ROOT_DIR=/path/to/openssl.

2. cmake .. -DLLVM_TARGETS_TO_BUILD="X86" -DCMAKE_INSTALL_PREFIX=... -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS=-I$OPENSSL_ROOT_DIR/include -DCMAKE_CXX_FLAGS=-I$OPENSSL_ROOT_DIR/include

Deprecated: Building LLVM and Clang using configure:

cd llvm

mkdir build

cd build

../configure --prefix=... --enable-optimized --enable-targets=x86,x86_64

make

make install

Options:

The release version is optimized. Switch --enable-optimized with --disable-optimized for a debug (10x slower!!) build.

For LTO:

1. use --with-binutils-include=<binutils 'include' path>

2. to install in your own prefix directory, make sure that both binutils and LLVM have the same prefix, then make install both toolchains to the same prefix.

3. also, copy or link from LLVMgold.so to prefix/lib/bfd-plugins/LLVMgold.so and make sure prefix/bin/ld points to ld.gold, not ld.bfd.

--with-built-clang instructs the test-suite to use the built version of clang rather than llvm-gcc.

For 64-bit targets, use --enable-targets=x86_64 or --enable-targets=x86,x86_64.

Note: On some systems the prefix path must be absolute: /home/myuser/multicompiler/install rather than ../install.

Options

For LTO, all -mllvm -option-here options should be translated to the form -Wl,--plugin-opt,-option-here and -flto added to the compilation (and potentially linking, if they are different) flags.

important: When using build systems that use libtool (e.g. Apache), you must include -flto in the compiler path (CC), rather than CFLAGS/LDFLAGS. Libtool (stupidly and silently) strips unrecognized options such as flto from the CFLAGS/LDFLAGS. Example of correct usage: export CC="/path/to/multicompiler/clang -flto"

General Options

-frandom-seed=# - Set the random seed to #.(The type of # is uint64_t, which means the range should be within 0 - (2^64-1))

For LTO: -Wl,--plugin-opt,-random-seed=#

Stack-layout randomization and reversal

-mllvm -shuffle-stack-frames - Enable stack-layout randomization.

-mllvm -reverse-stack-frames - Reverse layout of each stack frame.

-mllvm -stack-frame-random-seed=SEED - Distinct stack frame randomization seed. Overrides -frandom-seed (or -random-seed above) for this randomization (and stack frame padding).

Insert padding between stack frames

-mllvm -max-stack-pad-size=# - Enable inter-stack-frame padding of which the maximum size is #. (The preferable size is between 0 and 256)

-mllvm -stack-frame-random-seed=SEED - Distinct stack frame randomization seed. Overrides -frandom-seed (or -random-seed above) for this randomization (and stack frame shuffling).

Stack-element padding

Insert padding between elements in the unsafe stack.

-mllvm -stack-element-percentage=# - Percentage of elements in the unsafe stack prepended by paddings.

-mllvm -max-stack-element-pad-size=# - Maximum size of stack element padding (The preferable size is between 0 and 50).

-mllvm -stack-element-padding-random-seed=SEED - Distinct stack element padding randomization seed. Overrides -frandom-seed (or -random-seed above) for this randomization.

Function randomization (LTO req'd)

Using this transformation without LTO is possible but not recommended.

-mllvm -randomize-function-list - Enable function randomization.

Machine register randomization

-mllvm -randomize-machine-registers - Enable machine register randomization.

Safestack

-fsanitize=safe-stack - Enable Safestack. This feature places buffers and other "address-taken" variables on a separate stack to prevent stack smashing.

NOTE: this option should be passed to both compiler and linker as any other sanitizer flags.

Stack-to-heap promotion

-mllvm -stack-to-heap-promotion - Enable stack-to-heap promotion: this feature randomly promotes buffers in stack slots to heap.

-mllvm -stack-to-heap-percentage=# - Percentage of buffers to be promoted to heap.

The promoted slots are malloc'ed in the beginning and then free'd when function returns (performance concerns). In the current implementation, the promoted stack slots and their pointers are not remained in the stack.

TODO: 1) This currently does not promote buffers in safestack if it is used with SafeStack. 2) Promoting DynamicAllocas is currently not supported.

NOTE: This transformation should not be applied to signal handlers because it inserts async-signal-unsafe functions: malloc() and free(). To avoid this issue, we whitelist signal handlers defined in ATDSigHandlers.def.

Code-pointer protection (LTO req'd)

-fcode-pointer-protection - Replaces code pointers with pointers to trampolines. Prevents code pointers from leaking the code layout. Cookies are used to authenticate calls and returns through trampolines. Intended to be used in conjunction with X-only memory. Note: this feature works with stack unwinding but likely breaks C++ exception handling.

For LTO: `-Wl,--plugin-opt,-pointer-protection -Wl,--