ReFlex

ReFlex is a software-based system that provides remote access to Flash with performance nearly identical to local Flash access. ReFlex closely integrates network and storage processing in a dataplane kernel to achieve low latency and high throughput at low resource requirements. ReFlex uses a novel I/O scheduler to enforce tail latency and throughput service-level objectives (SLOs) for multiple clients sharing a remote Flash device.

ReFlex is an extension of the IX dataplace operating system. ReFlex uses Intel DPDK for fast packet processing and Intel SPDK for high performance access to NVMe Flash. There are two implementations of ReFlex currently available: a kernel implementation and a userspace implementation.

In the kernel implementation (available in the master branch of this repository), the ReFlex kernel runs as a guest OS in Linux and relies on the Dune kernel for memory management and direct access to hardware (e.g., NIC and NVMe Flash queues) from userspace applications. This is the original implementation of ReFlex, as presented in the paper published at ASPLOS'17.

In the userspace implementation (available in the userspace branch), network and storage processing is implemented in userspace and ReFlex uses the standard igb_uio module to bind a network device to a DPDK-provided network device driver. The userspace version of ReFlex does not require the Dune kernel module to be loaded. This means the userspace version of ReFlex is simpler to deploy.

Requirements for userspace version of ReFlex

ReFlex requires a NVMe Flash device and a network interface card supported by Intel DPDK. We have tested ReFlex with the Intel 82599 10GbE NIC and the following NVMe SSDs: Samsung PM1725 and Intel P3600. We have also tested ReFlex on Amazon Web Services EC2 instances i3.4xlarge (see end of setup instructions below for AWS EC2 instance setup).

ReFlex has been successfully tested on Ubuntu 16.04 LTS with kernel 4.4.0.

Note: ReFlex provides an efficient dataplane for remote access to Flash. To deploy ReFlex in a datacenter cluster, ReFlex should be combined with a control plane to manage Flash resources across machines and optimize the allocation of Flash IOPS and capacity.

Setup Instructions

There is currently no binary distribution of ReFlex. You will therefore need to compile the project from source, as described below.

Obtain ReFlex source code and fetch dependencies:

git clone https://github.com/stanford-mast/reflex.git
cd reflex
git checkout userspace
./deps/fetch-deps.sh

Install library dependencies:

sudo apt-get install libconfig-dev libnuma-dev libpciaccess-dev libaio-dev libevent-dev g++-multilib libcunit1-dev libssl-dev

Build the dependecies:

sudo chmod +r /boot/System.map-`uname -r`
make -sj64 -C deps/dpdk config T=x86_64-native-linuxapp-gcc
make -sj64 -C deps/dpdk
make -sj64 -C deps/dpdk install T=x86_64-native-linuxapp-gcc DESTDIR=deps/dpdk/x86_64-native-linuxapp-gcc
export REFLEX_HOME=`pwd`
cd deps/spdk
./configure --with-dpdk=$REFLEX_HOME/deps/dpdk/x86_64-native-linuxapp-gcc
make
cd ../..

Build ReFlex:
```
make -sj64
```

Set up the environment:

cp ix.conf.sample ix.conf
 # modify at least host_addr, gateway_addr, devices, and nvme_devices

sudo modprobe -r ixgbe
sudo modprobe -r nvme
sudo modprobe uio
sudo insmod deps/dpdk/build/kmod/igb_uio.ko
sudo deps/dpdk/usertools/dpdk-devbind.py --bind=igb_uio 0000:06:00.0   # insert device PCI address here!!! 

sudo deps/spdk/scripts/setup.sh

sudo sh -c 'for i in /sys/devices/system/node/node*/hugepages/hugepages-2048kB/nr_hugepages; do echo 4096 > $i; done'

Precondition the SSD and derive the request cost model:

It is necessary to precondition the SSD to acheive steady-state performance and reproducible results across runs. We recommend preconditioning the SSD with the following local Flash tests: write sequentially to the entire address space using 128KB requests, then write randomly to the device with 4KB requests until you see performance reach steady state. The random write test typically takes about 10 to 20 minutes, depending on the device model and capacity.

After preconditioning, you should derive the request cost model for your SSD:
```
cp sample.devmodel nvme_devname.devmodel
 # update request costs and token limits to match your SSD performance 
 # see instructions in comments of file sample.devmodel
 # in ix.conf file, update nvme_device_model=nvme_devname.devmodel
```
You may use any I/O load generation tool (e.g. fio) for preconditioning and request calibration tests. Note that if you use a Linux-based tool, you will need to reload the nvme kernel module for these tests (remember to unload it before running the ReFlex server).

For your convenience, we provide an open-loop, local Flash load generator based on the SPDK perf example application here. We modified the SPDK perf example application to report read and write percentile latencies. We also made the load generator open-loop, so you can sweep throughput by specifying a target IOPS instead of queue depth. See setup instructions for ReFlex users in the repository's README.

Instructions for setting up an AWS EC2 instance to run ReFlex

To run ReFlex on Amazon Web Servies EC2, you will need an instance type that supports Enhanced Networking (this means the instance has a DPDK-supported Enhanced Networking Adapter). The instance also needs to have NVMe Flash. Use an i3.4xlarge or larger instance with Ubuntu 16.04 Linux. Using a dedicated instance is strongly recommended for performance measurements.

You will need to setup a Virtual Private Cloud (VPC) for your instance(s) so that you can add Ethernet interfaces on the instance(s) running ReFlex. When you are ready to launch your ReFlex instance, in the "configure instance details" step, you need to add an extra network interface. This is required because you need to bind one interface to the userspace igb_uio driver for ReFlex/DPDK and you need one interface bound to the kernel NIC driver to SSH into your instance. In the "configure instance details" step, set the network for your instance to be your VPC network and use the subnet you created within your VPC. You can create the subnet from the VPC dashboard.

To connect to your instance, you should create an Elastic IP address and then associate it to the eth0 interface on your instance. Go to your VPC dashboard -> Elastic IPs -> Allocate new address. Next, associate this address to the eth0 interface on your ReFlex instance after you have obtained the interface ID and private IP address by hovering over the eth0 field in the description of your instance in the EC2 dashboard.

When you have successfully launched and connected to your instance, follow the ReFlex setup instructions (steps 1-6) above.

Running ReFlex

1. Run the ReFlex server:

sudo ./dp/ix

ReFlex runs one dataplane thread per CPU core. If you want to run multiple ReFlex threads (to support higher throughput), set the cpu list in ix.conf and add fdir rules to steer traffic identified by {dest IP, src IP, dest port} to a particular core.

Registering service level objectives (SLOs) for ReFlex tenants:

A tenant is a logical abstraction for accounting for and enforcing SLOs. ReFlex supports two types of tenants: latency-critical (LC) and best-effort (BE) tenants.
The current implementation of ReFlex requires tenant SLOs to be specified statically (before running ReFlex) in pp_accept() in apps/reflex_server.c. Each port ReFlex listens on can be associated with a separate SLO. The tenant should communicate with ReFlex using the destination port that corresponds to the appropriate SLO. This is a temporary implementation until there is proper client API support for a tenant to dynamically register SLOs with ReFlex.

As future work, a more elegant approach would be to i) implement a ReFlex control plane that listens on a dedicated admin port, ii) provide a client API for a tenant to register with ReFlex on this admin port and specify its SLO, and iii) provide a response from the ReFlex control plane to the tenant, indicating which port the tenant should use to communicate with the ReFlex data plane.

2. Run a ReFlex client:

There are several options for clients:

Run a Linux-based client that opens TCP connections to ReFlex and sends read/write reqs to logical blocks.
- This client option avoids the performance overheads of the filesystem and block layers in the client machine. However, the client is still subject to any latency or throughput inefficiencies of the networking layer in the Linux operating system.
On the client machine, setup the mutilate load generator. We provide a modified version of mutilate which supports the ReFlex block protocol.
```
git clone https://github.com/anakli/mutilate.git
cd mutilate
sudo apt-get install scons libevent-dev gengetopt libzmq-dev
scons
```
To achieve the best performance with a Linux client, we recommend tuning NIC settings and

Reflex

Install / Use

README