README

SLAMBench Release Candidate 1.2

Developed in the PAMELA project, EPSRC Programme Grant EP/K008730/1

What is it for?

A SLAM performance benchmark that combines a framework for quantifying quality-of-result with instrumentation of execution time and energy consumption.
It contains a KinectFusion (http://research.microsoft.com/pubs/155378/ismar2011.pdf) implementation in C++, OpenMP, OpenCL and CUDA (inspired by https://github.com/GerhardR).
It offers a platform for a broad spectrum of future research in jointly exploring the design space of algorithmic and implementation-level optimisations.
Target desktop, laptop, mobile and embedded platforms. Tested on Ubuntu, OS X and Android (on Android only the benchmark application has been ported, see later).

If you use SLAMBench in scientific publications, we would appreciate citations to the following paper (http://arxiv.org/abs/1410.2167):

L. Nardi, B. Bodin, M. Z. Zia, J. Mawer, A. Nisbet, P. H. J. Kelly, A. J. Davison, M. Luján, M. F. P. O’Boyle, G. Riley, N. Topham, and S. Furber. Introducing SLAMBench, a performance and accuracy benchmarking methodology for SLAM. In IEEE Intl. Conf. on Robotics and Automation (ICRA), May 2015. arXiv:1410.2167.

Bibtex entry:

#!latex

@inproceedings{Nardi2015,
    author={Nardi, Luigi and Bodin, Bruno and Zia, M. Zeeshan and Mawer, John and Nisbet, Andy and Kelly, Paul H. J. and Davison, Andrew J. and Luj\'an, Mikel and O'Boyle, Michael F. P. and Riley, Graham and Topham, Nigel and Furber, Steve},
    title = "{Introducing SLAMBench, a performance and accuracy benchmarking methodology for SLAM}",
    booktitle = "{IEEE Intl. Conf. on Robotics and Automation (ICRA)}",
    year = {2015},
    month = {May},
    NOTE = {arXiv:1410.2167}
    }

How do I get set up on Ubuntu?

If you want to set up for OS X go to the relevant section.

Dependencies

Required

TooN: maths library.
CMake 2.8+ : building tool.

Install TooN and CMake#####

#!shell
git clone git://github.com/edrosten/TooN.git
cd TooN
./configure
sudo make install
sudo apt-get install cmake

+(with Ubuntu, you might need to install the build-essential package using sudo apt-get update && sudo apt-get install build-essential)

Optional

OpenMP : for the OpenMP version
CUDA : for the CUDA version
OpenCL : for the OpenCL version (OpenCL 1.1 or greater)
OpenGL / GLUT : used by the graphical interface
OpenNI : for the live mode, and for oni2raw tool (which convert an OpenNI file to the SLAMBench internal format)
Freenect Drivers : In order to use the live mode.
PkgConfig / Qt5 (using OpenGL) : used by the Qt graphical interface (not fully required to get a graphical interface)
Python (numpy) : use by benchmarking scripts (mean, max, min functions)

Installation of Qt5 with an ARM board (ie. Arndale, ODROID,...)

On ARM board, the default release of Qt5 was compile using OpenEGL, to use the Qt interface, you will have to compile Qt :

#!
cd ~
wget http://download.qt-project.org/official_releases/qt/5.2/5.2.1/single/qt-everywhere-opensource-src-5.2.1.tar.gz
tar xzf qt-everywhere-opensource-src-5.2.1.tar.gz
cd ~/qt-everywhere-opensource-src-5.2.1
./configure -prefix ~/.local/qt/ -no-compile-examples  -confirm-license  -release   -nomake tests   -nomake examples
make
make install

Compilation of SLAMBench

Then simply build doing:

#!
make

To use qt, if you compile the source as explained above, you should need to specify the Qt install dir :

#!
CMAKE_PREFIX_PATH=~/.local/qt/ make

Running SLAMBench

The compilation builds 3 application modes which act like wrappers (the kernels are the same for all applications):

benchmark: terminal user interface mode for benchmarking purposes
main: GLUT GUI visualisation mode
qmain: Qt GUI visualisation mode

Each application mode is also declined in 4 different builds/implementations:

C++ (./build/kfusion/kfusion-main-cpp)
OpenMP (./build/kfusion/kfusion-main-openmp)
OpenCL (./build/kfusion/kfusion-main-cpp)
CUDA (./build/kfusion/kfusion-main-cuda)

The Makefile will automatically build the executable with satisfied dependencies, e.g. the OpenCL application version will be built only if the OpenCL tool kit is available on the system and so on for CUDA, OpenMP and Qt.

All application modes and implementations share the same set of arguments:

#!plain
-c  (--compute-size-ratio)       : default is 1   (same size)      
-d  (--dump-volume) <filename>   : Output volume file   
-f  (--fps)                      : default is 0
-i  (--input-file) <filename>    : Input camera file               
-k  (--camera)                   : default is defined by input     
-l  (--icp-threshold)        : default is 1e-05
-o  (--log-file) <filename>      : default is stdout               
-m  (--mu)                       : default is 0.1               
-p  (--init-pose)                : default is 0.5,0.5,0     
-q (--no-gui)                    : disable any gui used by the executable
-r  (--integration-rate)         : default is 1     
-s  (--volume-size)              : default is 2,2,2      
-t  (--tracking-rate)            : default is 1     
-v  (--volume-resolution)        : default is 256,256,256    
-y  (--pyramid-levels)           : default is 10,5,4 
-z  (--rendering-rate)   : default is 4

SLAMBench supports several input streams (how to use these inputs is described later):

ICL-NUIM dataset (http://www.doc.ic.ac.uk/~ahanda/VaFRIC/iclnuim.html)
RGB-D camera like Microsoft Kinect or other PrimeSense cameras using the OpenNI interface
OpenNI pre-recorded file
Raw format

1. benchmark mode

Use this mode for benchmarking proposes. The output is:

frame : ID number of the current frame
acquisition : input data acquisition elapsed time (file reading)
preprocessing : pre-processing elapsed time (includes kernels mm2meters bilateralFilter)
tracking : tracking elapsed time (includes kernels halfSample, depth2vertex vertex2normal, track, reduce and solve)
integration : integration elapsed time (includes kernel integrate)
raycast : raycast elapsed time (include kernel raycast)
rendering : rendering elapsed time (includes kernels renderDepth renderTrack and renderVolume)
computation : pre-processing + tracking + integration + raycast. This is the total elapsed time for processing a frame but not including the acquiring and the visualisation kernels
total : computation + acquisition + rendering. This is the total elapsed time for processing one frame (including the acquiring and the visualisation kernels)
X,Y,Z : estimation of the camera position (tracking result)
tracked : this boolean indicates if for the current frame we have not lost the tracking of the camera (1 = tracking, 0 = tracking lost)
integrated : this boolean indicates if the integration step occurs for the current frame (depending of the tracking result and of the integration rate)

How to use the benchmark mode with the ICL-NUIM dataset

SLAMBench provides an interface to the ICL-NUIM dataset. This enables the accuracy evaluation on a SLAM implementation via the ICL-NUIM ground truth. ICL-NUIM provides 4 trajectories, we pick trajectory 2 and show how to use the dataset (for the download of each trajectory we recommend 2 GB of space available on the system):

#!plain
mkdir living_room_traj2_loop
cd living_room_traj2_loop
wget http://www.doc.ic.ac.uk/~ahanda/living_room_traj2_loop.tgz
tar xzf living_room_traj2_loop.tgz
cd ..

You can use the ICL-NUIM dataset in its native format or in a RAW format (with the latter acquiring speed increases). RAW is the format to be used for benchmarking purposes, to generate the RAW file:

#!plain
./build/kfusion/thirdparty/scene2raw living_room_traj2_loop living_room_traj2_loop.raw

Run SLAMBench:

#!plain
./build/kfusion/kfusion-benchmark-cuda -i living_room_traj2_loop.raw  -s 4.8 -p 0.34,0.5,0.24 -z 4 -c 2 -r 1 -k 481.2,480,320,240  > benchmark.log

You can replace cuda by openmp, opencl or cpp.

In order to check the accuracy of your tracking compared to the ground truth trajectory, first download the ground truth trajectory file:

#!plain
wget http://www.doc.ic.ac.uk/~ahanda/VaFRIC/livingRoom2.gt.freiburg

And then use the following tool:

#!plain
./kfusion/thirdparty/checkPos.py benchmark.log livingRoom2.gt.freiburg 
Get slambench data. 
slambench result        : 882 positions.
NUIM  result        : 880 positions.
Working position is : 880
Runtimes are in seconds and the absolute trajectory error (ATE) is in meters.
The ATE measure accuracy, check this number to see how precise your computation is.
Acceptable values are in the range of few centimeters.

            tracking 	Min : 0.005833 	Max : 0.046185 	Mean : 0.020473 	Total : 18.05721755
         integration 	Min : 0.003629 	Max : 0.041839 	Mean : 0.021960 	Total : 19.36882799
           rendering 	Min : 0.018242 	Max : 0.022144 	Mean : 0.018486 	Total : 16.30500085
       preprocessing 	Min : 0.000633 	Max : 0.002064 	Mean : 0.000719 	Total : 0.63432674
         computation 	Min : 0.010156 	Max : 0.075946 	Mean : 0.043152 	Total : 38.06037227
           total     	Min : 0.028520 	Max : 0.094302 	Mean : 0.061724 	Total : 54.44080794
                 ATE 	Min : 0.000000 	Max : 0.044235 	Mean : 0.018392 	Total : 16.

Slambench1

Install / Use

README

README

What is it for?

How do I get set up on Ubuntu?

Dependencies

Required

Install TooN and CMake#####

Optional

Installation of Qt5 with an ARM board (ie. Arndale, ODROID,...)

Compilation of SLAMBench

Running SLAMBench

1. benchmark mode

How to use the benchmark mode with the ICL-NUIM dataset