GMMap: Memory-Efficient Continuous Occupancy Map Using Gaussian Mixture Model

👋 Questions & Support: For inquiries about the paper or code, please use GitHub Issues rather than email. This ensures that our responses are documented and visible to the entire community.

Overview

Energy consumption of memory accesses dominates the compute energy in energy-constrained robots which require a compact 3D map of the environment to achieve autonomy. Recent mapping frameworks only focused on reducing the map size while incurring significant memory usage during map construction due to multi-pass processing of each depth image. In this work, we present a memory-efficient continuous occupancy map, named GMMap, that accurately models the 3D environment using a Gaussian Mixture Model (GMM). Memory-efficient GMMap construction is enabled by the single-pass compression of depth images into local GMMs which are directly fused together into a globally-consistent map. By extending Gaussian Mixture Regression to model unexplored regions, occupancy probability is directly computed from Gaussians. Using a low-power ARM Cortex A57 CPU, GMMap can be constructed in real-time at up to 60 images per second. Compared with prior works, GMMap maintains high accuracy while reducing the map size by at least 56%, memory overhead by at least 88%, DRAM access by at least 78%, and energy consumption by at least 69%. Thus, GMMap enables real-time 3D mapping on energy-constrained robots.

Publication & Presentation

You can find our Transactions on Robotics (T-RO) paper on IEEE here and arXiv here.

Our presentation can be found here.

The figure below visualizes a room environment overlaid with the following mapping frameworks: (a) GMMap (occupied Gaussians), (b) GMMap (occupied voxels from uniform voxel-grid sampling), (c) NDT-OM (occupied Gaussians), (d) BGKOctoMap-L (occupied voxels), and (e) OctoMap (occupied voxels). For both GMMap and NDT-OM, the surface boundaries of the obstacles are smooth and often extend beyond their ellipsoidal wireframes (e.g., compare (a) and (b) for GMMap).

If you find our work useful, please cite it as follows:

@ARTICLE{GMMap,
  author={Li, Peter Zhi Xuan and Karaman, Sertac and Sze, Vivienne},
  journal={IEEE Transactions on Robotics}, 
  title={GMMap: Memory-Efficient Continuous Occupancy Map Using Gaussian Mixture Model}, 
  year={2024},
  volume={40},
  number={},
  pages={1339-1355},
  keywords={Memory management;Robot sensing systems;Robots;Three-dimensional displays;Random access memory;Image coding;Computational modeling;Mapping;memory efficiency;RGB-D perception;sensor fusion},
  doi={10.1109/TRO.2023.3348305}}

Barebone Deployment (Mapping Only)

To build the library without any visualization and dataset capabilities, install eigen and fmt from source a package manager.

When building the library to use the CPU only, we do the following

cd gmm_mapping
mkdir build
cd build
cmake -DGMMAP_BUILD_CUDA=OFF -DGMMAP_BUILD_EXAMPLES=OFF ..
make

If GPU (CUDA) acceleration is required, we do the following

mkdir build
cd build
cmake -DGMMAP_BUILD_CUDA=ON -DGMMAP_BUILD_EXAMPLES=OFF ..
make

This should create a static mapping library for you to link your project (if you have your own custom visualizer and dataset loader).

Dependencies (Mapping & Visualization)

Please install the following dependencies:

1. Compile Open3D from source.

   git clone --depth 1 --branch v0.18.0 https://github.com/isl-org/Open3D
   cd Open3D
   mkdir build
   cd build
   cmake -DGLIBCXX_USE_CXX11_ABI=ON ..
   sudo make install -j$(nproc)
   

   • Note: Open3D's C++ APIs can change and are not guaranteed to be backward compatible. If you install a newer version of Open3D, you might need to modify the code presented in this repository!
   • Since 3rd party libraries are installed during Open3D compilation process, we need sudo make in order to ensure that all libraries can be linked appropriately.
   • When building with Apple Silicon, ensure that python executable exists (or properly linked)!
   • If you need CUDA accelerated visualization, build with flag -DBUILD_CUDA_MODULE=ON instead.
   • On Apple Silicon, install OpenMP via brew and build Open3D with the following options:

     cmake -DGLIBCXX_USE_CXX11_ABI=ON -DWITH_OPENMP=ON -DBUILD_PYTHON_MODULE=OFF \
     -DOpenMP_C_FLAGS=-fopenmp \
     -DOpenMP_CXX_FLAGS=-fopenmp \
     -DOpenMP_C_LIB_NAMES="libomp" \
     -DOpenMP_CXX_LIB_NAMES="libomp" \
     -DOpenMP_libomp_LIBRARY="/opt/homebrew/opt/libomp/lib/libomp.dylib" \
     -DOpenMP_CXX_FLAGS="-Xpreprocessor -fopenmp -I/opt/homebrew/opt/libomp/include" \
     -DOpenMP_CXX_LIB_NAMES="libomp" \
     -DOpenMP_C_FLAGS="-Xpreprocessor -fopenmp -I/opt/homebrew/opt/libomp/include" ..
     

2. Install additional dependencies (from source or a package manager): eigen, jsoncpp.

3. Install additional dependences (from source to get the latest version): fmt, tabulate.

Additional notes for users with Apple Silicon

1. Do not install fmt before Open3D
2. Install gcc via brew. Then, change the LIBS_OPT variable in CMakeLists.txt to ensure that the library libguadmath.a is correctly located.
3. To ensure that OpenMP is supported, install llvm via brew and use its clang to compile this library (i.e., Do not use clang from XCode).

Build Instruction (Mapping & Visualization)

First, clone the repository as follows:

git clone https://github.com/mit-lean/GMMap.git

When building the library to use the CPU only, we do the following

cd gmm_mapping
mkdir build
cd build
cmake -DGMMAP_BUILD_CUDA=OFF -DGMMAP_BUILD_EXAMPLES=ON ..
make

If GPU (CUDA) acceleration is required, we do the following

mkdir build
cd build
cmake -DGMMAP_BUILD_CUDA=ON -DGMMAP_BUILD_EXAMPLES=ON ..
make

Adopting GMMap to Your Applications

All the GMMap parameters are read from a .json file. Examples of such files are located in the indoor and outdoor directory. Please refer to the instructions here for tips and explanations of each parameter. We also provide several examples in the following section to assist you.

Example Applications

Firstly, make sure to change the location of the dataset in the dataset_path::location of the *.json file within the indoor and outdoor directory. Note that this location is defined relative to the $HOME directory of your OS. All other directories are defined relative to the current directory with the executable. Please refer to our dataset utility for how the dataset is loaded. For your convenience, we included some sequences from TUM and Tartanair that you can download here.

Map Visualization

When mapping using CPU only, we run the visualization example for the Room sequence in the TUM dataset as follows:

./GMMMapVisualization tum ../example/indoor 0 3000 0 --device CPU:0

When mapping with CPU and GPU, we run the visualization example for the same sequence as follows:

./GMMMapCudaVisualization tum ../example/indoor 0 3000 --device CPU:0

To learn more about the functionalities of the command-line arguements, read the source code in the example folder. In addition, feel free to checkout the GUI options. Depending on your hardware, interacting with GUI too rapidly may cause it to crash. Thus, we recommend the following for large environment

1. Do not drag the sliders. Just click at the desired location.
2. The visualizer and GMMap construction are occuring on separate threads. Since the throughput of GMMap is very high, the visualizer is not able to keep up with the geometry updates and can crash. To prevent this, we provide Update Interval and Visualization Latency slider in the Reconstruction Settings panel. Updated geometries of the constructed map will be transferred to the visualizer in batches at a certain frame interval defined by the Update Interval slider. In addition, a small latency in ms is defined in Visualization Latency and is added after each image to slow down the GMMap construction. Feel free to play with these settings so that the visualizer run successfully on your hardware.
3. The throughput statistics displayed in the upper right-hand panel can be lower than expected because the visualizer "steals" significant CPU resources from map construction. Please run GMMap in headless mode instead!

Headless Mode

To accurately obtain the throughput of the algorithm, please run the mapping algorithm without the GUI in headless mode.

When using the CPU only, we run the headless example for the Room sequence in the TUM dataset as follows:

./GMMMapNoViz tum ../example/indoor 0 3000

You should expect to see some statistics summarizing the throughput and map size in the terminal, such as

Obstacle clusters: 1021
Free clusters: 273
Total cluster size: 111.20KB
Rtree size: 61.45KB,
Total Rtree nodes: 1573
GMM Clustering FPS: 470.336
GMM Fusion FPS: 2209.384
GMM Mapping FPS: 387.784
Map Size: 172.65KB

When mapping with CPU and GPU, we run the headless example for the same sequence as follows:

./GMMMapCudaNoViz tum ../example/indoor 0 3000

Path Planning

We also provide an example on how to use GMMap in a path planning scenario for the TUM Room sequence. In particular, we i) generate a map and then save it, ii) load the saved map, iii) c