<img src="assets/naruto.png" width="30" height="30"> <img src="assets/naruto.png" width="30" height="30"> <img src="assets/naruto.png" width="30" height="30"> NARUTO <img src="assets/naruto.png" width="30" height="30"> <img src="assets/naruto.png" width="30" height="30"> <img src="assets/naruto.png" width="30" height="30">

Neural Active Reconstruction from Uncertain Target Observations (CVPR-2024)

This is an official implementation of the paper "NARUTO: Neural Active Reconstruction from Uncertain Target Observations".

Ziyue Feng*†1,2, Huangying Zhan*†‡1, Zheng Chen†1,3, Qingan Yan1, Xiangyu Xu1, Changjiang Cai1, Bing Li2, Qilun Zhu2, Yi Xu1

1 OPPO US Research Center, 2 Clemson University 3 Indiana University

* Co-first authors with equal contribution † Work done as an intern at OPPO US Research Center ‡ Corresponding author

📁 Repo Structure
🛠️ Installation
💿 Dataset Preparation
🏃‍♂️ Running NARUTO
🔎 Evaluation
🎨 Run on Customized Scenes
📜 License
🤝 Acknowledgement
📖 Citation

<h2 id="repo-structure"> 📁 Repo Structure </h2>

# Main directory
├── NARUTO (ROOT)
│   ├── assets                                      # README assets
│   ├── configs                                     # experiment configs
│   ├── data                                        # dataset
│   │   └── MP3D                                    # Matterport3D for Habitat data
│   │   └── mp3d_data                               # Matterport3D raw Dataset
│   │   └── Replica                                 # Replica SLAM Dataset
│   │   └── replica_v1                              # Replica Dataset v1
│   ├── envs                                        # environment installation 
│   ├── results                                     # experiment results
│   ├── scripts                                     # scripts
│   │   └── data                                    # data related scripts
│   │   └── evaluation                              # evaluation related scripts
│   │   └── installation                            # installation related scripts
│   │   └── naruto                                  # running NARUTO scripts
│   ├── src                                         # source code
│   │   └── data                                    # data code
│   │   └── evaluation                              # evaluation code
│   │   └── layers                                  # pytorch layers
│   │   └── naruto                                  # NARUTO framework code
│   │   └── planning                                # planning code
│   │   └── simulator                               # simulator code
│   │   └── slam                                    # SLAM code
│   │   └── utils                                   # utility code
│   │   └── visualization                           # visualization code
│   ├── third_parties                               # third_parties
│   │   └── coslam                                  # CoSLAM 
│   │   └── habitat_sim                             # habitat-sim
│   │   └── neural_slam_eval                        # evaluation tool


# Data structure
├── data                                            # dataset dir
│   ├── MP3D                                        # Matterport3D data
│   │   └── v1
│   │       └── tasks
│   │           └── mp3d_habitat
│   │               ├── 1LXtFkjw3qL
│   │               └── ...
│   ├── replica_v1                                  # Replica-Dataset
│   │   └── apartment_0
│   │       └── habitat
│   │           └── replicaSDK_stage.stage_config.json
│   │   └── ...
│   ├── Replica                                     # Replica SLAM Dataset
│   │   └── office_0
│   │   └── ...

# Configuration structure
├── configs                                         # configuration dir
│   ├── default.py                                  # Default overall configuration
│   ├── MP3D                                        # Matterport3D 
│   │   └── mp3d_coslam.yaml                        # CoSLAM default configuration for MP3D
│   │   └── {SCENE}
│   │       └── {EXP}.py                            # experiment-specific overall configuraiton, inherit from default.py
│   │       └── coslam.yaml                         # scene-specific CoSLAM configuration, inherit from mp3d_coslam.yaml
│   │       └── habitat.py                          # scene-specific HabitatSim configuration
│   │   └── ...
│   ├── Replica                                     # Replica
│   │   └── ...

NOTE: default.py/{EXP}.py has the highest priority that can override configurations in other config files (e.g. mp3d_coslam.yaml/coslam.yaml, habitat.py)

# Result structure
├── results                                         # result dir
│   ├── MP3D                                        # Matterport3D result
│   │   └── GdvgFV5R1Z5
│   │       └── {EXPERIMENT_SETUP}
│   │           └── run_{COUNT}
│   │               └── eval_result.txt
│   │               └── coslam
│   │                   └── checkpoint
│   │                   └── mesh
│   │   └── ...
│   ├── Replica                                     # Replica result
│   │   └── office_0
│   │       └── {EXPERIMENT_SETUP}
│   │           └── run_{COUNT}
│   │               └── eval_result.txt
│   │               └── coslam
│   │                   └── checkpoint
│   │                   └── mesh
│   │   └── ...

<h2 id="installation"> 🛠️ Installation </h2>

Install NARUTO

# Clone the repo with the required third parties.
git clone --recursive https://github.com/oppo-us-research/NARUTO.git

# We assume ROOT as the project directory.
cd NARUTO
ROOT=${PWD}

In this repo, we provide two types of environement installations: Docker and Anaconda.

Users can optionally install one of them. The installation process includes:

installation of Habitat-Sim, where we install our updated Habitat-Sim, where the geometry compilation is updated.
installation of Co-SLAM, which is used as our mapping backbone.
installation of other packages required to run NARUTO.

<img src="assets/docker_logo.png" width="20" height="20"> [Optional 1] Docker Environment

Follow the steps to install the Docker environment:

# Build Docker image
bash scripts/installation/docker_env/build.sh

# Run Docker container
bash scripts/installation/docker_env/run.sh

# Activate conda env in Docker Env
source activate naruto

# Install tinycudann as required in Co-SLAM
# We try to include this installation while building Docker but it fails somehow. 
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

<img src="assets/anaconda_logo.png" width="15" height="15"> [Optional 2] Conda Environment

Follow the steps to install the conda environment

# Build conda environment
bash scripts/installation/conda_env/build.sh

# Activate conda env
source activate naruto

<h2 id="dataset-preparation"> 💿 Dataset Preparation </h2>

<img src="assets/meta_logo.png" width="25" height="15"> Replica Data

Follow the steps to download Replica Dataset.

# Download Replica data and save as data/replica_v1.
# This process can take a while.
bash scripts/data/replica_download.sh data/replica_v1

# Once the donwload is completed, create modified Habitat Simulator configs that adjust the coordinate system direction.
# P.S. we adjust the config so the coordinates matches with the mesh coordinates.
bash scripts/data/replica_update.sh data/replica_v1

[Optional] We also use Replica Data (for SLAM) for some tasks, e.g. passive mapping / initialize the starting position.

# Download Replica (SLAM) Data and save as data/Replica
bash scripts/data/replica_slam_download.sh

<img src="assets/matterport_logo.png" width="15" height="15"> Matterport3D

To download Matterport3D dataset, please refer to the instruction in Matterport3D.

The download script is not included here as there is a Term of Use agreement for using Matterport3D data.

However, our method does not require the full Matterport3D dataset. Users can download the data related to the task habitat only.

# Example use of the Matterport3D download script:
python download

NARUTO

Install / Use

README