Kintinuous

Real-time dense visual SLAM system capable of producing high quality globally consistent point and mesh reconstructions over hundreds of metres in real-time with only a low-cost commodity RGB-D sensor.

Related Publications

Please cite the most appropriate of these works (in order of our preference) if you make use of our system in any of your own endeavors:

• Real-time Large Scale Dense RGB-D SLAM with Volumetric Fusion, T. Whelan, M. Kaess, H. Johannsson, M.F. Fallon, J. J. Leonard and J.B. McDonald, IJRR '14
• Deformation-based Loop Closure for Large Scale Dense RGB-D SLAM, T. Whelan, M. Kaess, J.J. Leonard, and J.B. McDonald, IROS '13
• Robust Real-Time Visual Odometry for Dense RGB-D Mapping, T. Whelan, H. Johannsson, M. Kaess, J.J. Leonard, and J.B. McDonald, ICRA '13
• Kintinuous: Spatially Extended KinectFusion, T. Whelan, M. Kaess, M.F. Fallon, H. Johannsson, J. J. Leonard and J.B. McDonald, RSS RGB-D Workshop '12
• A method and system for mapping an environment, T. Whelan, M. Kaess, J.J. Leonard and J.B. McDonald, US 9412173 B2

1. What do I need to build it?

• Ubuntu 14.04, 15.04 or 16.04 (Though many other linux distros will work fine)
• CMake
• OpenGL
• CUDA >= 7.0
• OpenNI2
• SuiteSparse
• Eigen
• Boost
• zlib
• libjpeg
• OpenCV
• DLib @ 330bdc10576f6bcb55e0bd85cd5296f39ba8811a
• DBoW2 @ 4a6eed2b3ae35ed6837c8ba226b55b30faaf419d
• DLoopDetector @ 84bfc56320371bed97cab8aad3aa9561ca931d3f
• iSAM
• PCL
• Pangolin

Firstly, add nVidia's official CUDA repository to your apt sources, then run the following command to pull in most dependencies from the official repos:

sudo apt-get install -y cmake-qt-gui git build-essential libusb-1.0-0-dev libudev-dev openjdk-7-jdk freeglut3-dev python-vtk libvtk-java libglew-dev cuda-7-5 libsuitesparse-dev openexr

This is where things get really bad. Due to libraries constantly changing their APIs and includes, creating build processes that actually last for more than a few months between a couple of Ubuntu versions is extremely difficult. Below are separate instructions for Ubuntu 14.04, 15.04 and 16.04.

14.04

Install PCL 1.7 from this PPA:

sudo add-apt-repository -y ppa:v-launchpad-jochen-sprickerhof-de/pcl
sudo apt-get update
sudo apt-get install -y libpcl-all

Then install OpenCV, DLib, DBoW2, DLoopDetector, iSAM and Pangolin from source, in this order.

Why do you have to install OpenCV from source? Because the version in the Ubuntu repos doesn't have the nonfree module, required for SURF descriptors used in the DBoW2. Also, it is strongly recommended you build OpenCV with the following options (in particular, building it with Qt5 might introduce a lot of pain):

cmake -D BUILD_NEW_PYTHON_SUPPORT=OFF -D WITH_OPENCL=OFF -D WITH_OPENMP=ON -D INSTALL_C_EXAMPLES=OFF -D BUILD_DOCS=OFF -D BUILD_EXAMPLES=OFF -D WITH_QT=OFF -D WITH_OPENGL=OFF -D WITH_VTK=OFF -D BUILD_PERF_TESTS=OFF -D BUILD_TESTS=OFF -D WITH_CUDA=OFF -D BUILD_opencv_gpu=OFF ..

If you have trouble building Pangolin, disable ffmpeg support using the following CMake command:

cmake .. -DAVFORMAT_INCLUDE_DIR=""

Once finished you'll have everything you need to build Kintinuous.

15.04

On 15.04 PCL is in the official repos:

sudo apt-get install -y libpcl-dev yasm libvtk5-qt4-dev

The version of PCL in the 15.04 repos does not contain OpenNI2. You should build the Occipital maintained version, available here.

As usual, ffmpeg screws things up for everyone in 15.04. You need to build ffmpeg yourself otherwise OpenCV will fail to build. Why do you have to install OpenCV from source? Because the version in the Ubuntu repos doesn't have the nonfree module, required for SURF descriptors used in the DBoW2. Why can't you use OpenCV3? Because DBoW will fail to build. Why can't you disable ffmpeg in OpenCV's build? Because DBoW will fail if the video module isn't built. Build and install ffmpeg as follows:

git clone git://source.ffmpeg.org/ffmpeg.git
cd ffmpeg/
git reset --hard cee7acfcfc1bc806044ff35ff7ec7b64528f99b1
./configure --enable-shared
make -j8
sudo make install
sudo ldconfig

Then build OpenCV with the following options:

cmake -D BUILD_NEW_PYTHON_SUPPORT=OFF -D WITH_OPENCL=OFF -D WITH_OPENMP=ON -D INSTALL_C_EXAMPLES=OFF -D BUILD_DOCS=OFF -D BUILD_EXAMPLES=OFF -D WITH_QT=OFF -D WITH_OPENGL=OFF -D WITH_VTK=OFF -D BUILD_PERF_TESTS=OFF -D BUILD_TESTS=OFF -D WITH_CUDA=OFF -D BUILD_opencv_gpu=OFF ..

Finally, build DLib, DBoW2, DLoopDetector, iSAM and Pangolin from source, in this order. If you have trouble building Pangolin, disable ffmpeg support using the following CMake command:

cmake .. -DAVFORMAT_INCLUDE_DIR=""

Afterwards, you will be able to build Kintinuous.

16.04

The instructions for 15.04 will work fine except you can't use the apt version of PCL because someone screwed up vtk. So ensure you purge/remove it completely if you have it installed from apt, then build it manually;

sudo apt-get install g++ cmake cmake-gui doxygen mpi-default-dev openmpi-bin openmpi-common libflann-dev libeigen3-dev libboost-all-dev libvtk5-qt4-dev libvtk6.2 libvtk5-dev libqhull* libusb-dev libgtest-dev git-core freeglut3-dev pkg-config build-essential libxmu-dev libxi-dev libusb-1.0-0-dev graphviz mono-complete qt-sdk openjdk-7-jdk openjdk-7-jre
git clone https://github.com/PointCloudLibrary/pcl.git
cd pcl
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_GPU=OFF -DBUILD_apps=OFF -DBUILD_examples=OFF ..
make -j8
sudo make install
sudo ldconfig

Afterwards, Kintinuous should link without any problems.

2. Is there an easier way to build it?

Understandably, building all of the dependencies seems quite complicated. If you run the build.sh script on a fresh clean install of Ubuntu 14.04 or 15.04, enter your password for sudo a few times and wait a few minutes all dependencies will get downloaded and installed and it should build everything correctly. This has not been tested on anything but fresh installs, so I would advise using it with caution if you already have some of the dependencies installed.

3. How do I use it?

There are four build targets:

• libutil.a is a small set of utility classes.
• libfrontend.a is the main tracking and fusion component.
• libbackend.a contains the triangulation, loop closure and deformation components.
• Kintinuous is an executable GUI used to run the system.

The GUI (Kintinuous) can take a bunch of parameters when launching it from the command line. They are as follows:

• -c <calibration> : Loads a camera calibration file specified by either:

• A depth_intrinsics matrix in OpenCV format (ending .yml, .xml), or
• A text file containing [fx fy cx cy] or [fx fy cx cy w h]

• -l <logfile> : Processes the specified .klg log file.
• -v <vocab> : Loads DBoW vocabulary file.
• -p <poses> : Loads ground truth poses to use instead of estimated pose.
• -gpu <gpu> : Sets which GPU should be used by CUDA.
• -n <number> : Number of frames to process.
• -t <threshold> : Voxel threshold for volume shifting (default 14).
• -cw <weight> : Removes voxels below this threshold when extracting slices (default 8).
• -lt <throttle> : Disallow loop closures within this time period of the last (default 30s).
• -s <size> : Size of the fusion volume (default 6m).
• -dg <sampling> : Rate of pose sampling for deformation (default 0.8m).
• -il <inliers> : Inlier threshold for RANSAC (default 0.35).
• -it <isam> : Residual threshold for pose graph optimisation (default 10).
• -sm : Static mode (disable volume shifting).
• -f : Flip RGB/BGR.
• -od : Perform online deformation (required for loop closure).
• -m : Enable mesh generation.
• -no : Disable overlap of extracted slices.
• -nos : Remove overlap when saving map.
• -r : Use RGB tracking only.
• -ri : Use combined ICP+RGB tracking.
• -d : Enable dynamic cube positioning.
• -dc : Disable color weighting by angle.
• -fl : Subsample pose graph for faster loop closure.
• -fod : Enable fast odometry.

Essentially by default ./Kintinuous will try run off an attached ASUS sensor live. You can provide a .klg log file instead with the -l parameter. You can capture .klg format logs using either Logger1 or Logger2.