InstantSplat

<h2 align="center"> <a href="https://arxiv.org/abs/2403.20309">InstantSplat: Sparse-view <a href="https://arxiv.org/abs/2403.20309"> Gaussian Splatting in Seconds </a> <h5 align="center">

</h5> <div align="center"> This repository is the official implementation of InstantSplat, a sparse-view framework for large-scale scene reconstruction method using Gaussian Splatting. InstantSplat supports 3D-GS, 2D-GS, and Mip-Splatting. </div> <br>

Table of Contents

• Table of Contents
• Free-view Rendering
• TODO List
• Get Started
  • Installation
  • Usage
• Acknowledgement
• Citation

Free-view Rendering

https://github.com/zhiwenfan/zhiwenfan.github.io/assets/34684115/748ae0de-8186-477a-bab3-3bed80362ad7

TODO List

• [x] Support 2D-GS
• [ ] Long sequence cross window alignment
• [ ] Support Mip-Splatting

Get Started

Installation

1. Clone InstantSplat and download pre-trained model.

git clone --recursive https://github.com/NVlabs/InstantSplat.git
cd InstantSplat
mkdir -p mast3r/checkpoints/
wget https://download.europe.naverlabs.com/ComputerVision/MASt3R/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth -P mast3r/checkpoints/

2. Create the environment (or use pre-built docker), here we show an example using conda.

conda create -n instantsplat python=3.10.13 cmake=3.14.0 -y
conda activate instantsplat
conda install pytorch torchvision pytorch-cuda=12.1 -c pytorch -c nvidia  # use the correct version of cuda for your system
pip install -r requirements.txt
pip install submodules/simple-knn
pip install submodules/diff-gaussian-rasterization
pip install submodules/fused-ssim

1. Optional but highly suggested, compile the cuda kernels for RoPE (as in CroCo v2).

# DUST3R relies on RoPE positional embeddings for which you can compile some cuda kernels for faster runtime.
cd croco/models/curope/
python setup.py build_ext --inplace

Alternative: use the pre-built docker image: pytorch/pytorch:2.1.2-cuda11.8-cudnn8-devel

docker pull dockerzhiwen/instantsplat_public:2.0

if docker failed to produce reasonable results, try Installation step again within the docker.

Usage

1. Data preparation (download our pre-processed data from: Hugging Face or Google Drive)

  cd <data_path>
  # then do whatever data preparation

1. Command

  # InstantSplat train and output video (no GT reference, render by interpolation) using the following command.
  # Users can place their data in the 'assets/examples/<scene_name>/images' folder and run the following command directly.
  bash scripts/run_infer.sh

  # InstantSplat train and evaluate (with GT reference) using the following command.
  bash scripts/run_eval.sh

Acknowledgement

This work is built on many amazing research works and open-source projects, thanks a lot to all the authors for sharing!

• Gaussian-Splatting and diff-gaussian-rasterization
• DUSt3R

Citation

If you find our work useful in your research, please consider giving a star :star: and citing the following paper :pencil:.

@misc{fan2024instantsplat,
        title={InstantSplat: Sparse-view Gaussian Splatting in Seconds},
        author={Zhiwen Fan and Kairun Wen and Wenyan Cong and Kevin Wang and Jian Zhang and Xinghao Ding and Danfei Xu and Boris Ivanovic and Marco Pavone and Georgios Pavlakos and Zhangyang Wang and Yue Wang},
        year={2024},
        eprint={2403.20309},
        archivePrefix={arXiv},
        primaryClass={cs.CV}
      }