Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis

CVPR 2024

Project Page | Paper | Video | Viewer & Pre-Trained Models

This is an official implementation of the paper "Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis".</br> Zhan Li^1,2, Zhang Chen^1,†, Zhong Li^1,†, Yi Xu¹ </br> ¹ OPPO US Research Center, ² Portland State University </br> ^† Corresponding authors </br>

<img src="assets/output.gif" width="100%"/></br>

Updates and News

• Jun 16, 2024: Added fully fused mlp for testing ours-full models on Technicolor and Neural 3D dataset (40 FPS improvement compared to paper).
• Jun 13, 2024: Fixed minors for reproducity on the scenes coffee_martini and flame_salmon_1 (~ 0.1 PSNR).
• Jun 9, 2024 : Supported lazy loading and ground truth image as int8 in GPU.
• Dec 28, 2023: Paper and Code are released.

Table of Contents

1. Installation
2. Preprocess Datasets
3. Training
4. Testing
5. Real-Time Viewer
6. Creating Your Gaussians
7. License Infomration
8. Acknowledgement
9. Citations

Installation

Windows users with WSL2 :

Please first refer to here to install the WSL2 system (Windows Subsystem for Linux 2) and install dependencies inside WSL2. Then you can set up our repo inside the Linux sub-system same as other Linux users.

Linux users :

Clone the source code of this repo.

git clone https://github.com/oppo-us-research/SpacetimeGaussians.git --recursive
cd SpacetimeGaussians

Then run the following command to install the environments with conda. Note we will create two environments, one for preprocessing with colmap (colmapenv) and one for training and testing (feature_splatting). Training, testing and preprocessing have been tested on Ubuntu 20.04. </br>

bash script/setup.sh

Note that you may need to manually install the following packages if you encounter errors during the installation of the above command. </br>

conda activate feature_splatting
pip install thirdparty/gaussian_splatting/submodules/gaussian_rasterization_ch9
pip install thirdparty/gaussian_splatting/submodules/gaussian_rasterization_ch3
pip install thirdparty/gaussian_splatting/submodules/forward_lite
pip install thirdparty/gaussian_splatting/submodules/forward_full

Processing Datasets

Note, our paper uses the sparse points that follow 3DGS. Our per frame SfM points only use point_triangulator in Colmap instead of dense points.

Neural 3D Dataset

Download the dataset from here. After downloading the dataset, you can run the following command to preprocess the dataset. </br>

conda activate colmapenv
python script/pre_n3d.py --videopath <location>/<scene>

<location> is the path to the dataset root folder, and <scene> is the name of a scene in the dataset. </br>

• For example if you put the dataset at /home/Neural3D, and want to preprocess the cook_spinach scene, you can run the following command

conda activate colmapenv
python script/pre_n3d.py --videopath /home/Neural3D/cook_spinach/

Our codebase expects the following directory structure for the Neural 3D Dataset after preprocessing:

<location>
|---cook_spinach
|   |---colmap_<0>
|   |---colmap_<...>
|   |---colmap_<299>
|---flame_salmon1

Technicolor Dataset

Please reach out to the authors of the paper "Dataset and Pipeline for Multi-View Light-Field Video" for access to the Technicolor dataset. </br> Our codebase expects the following directory structure for this dataset before preprocessing:

<location>
|---Fabien
|   |---Fabien_undist_<00257>_<08>.png
|   |---Fabien_undist_<.....>_<..>.png
|---Birthday

Then run the following command to preprocess the dataset. </br>

conda activate colmapenv
python script/pre_technicolor.py --videopath <location>/<scene>

Google Immersive Dataset

Download the dataset from here. After downloading and unzip the dataset, you can run the following command to preprocess the dataset. </br>

conda activate colmapenv
python script/pre_immersive_distorted.py --videopath <location>/<scene>
python script/pre_immersive_undistorted.py --videopath <location>/<scene>

<location> is the path to the dataset root folder, and <scene> is the name of a scene in the dataset. Please rename the orginal file to the name list Immersivesevenin here

• For example if you put the dataset at /home/immersive, and want to preprocess the 02_Flames scene, you can run the following command

conda activate colmapenv
python script/pre_immersive_distorted.py --videopath /home/immersive/02_Flames/

1. Our codebase expects the following directory structure for immersive dataset before preprocessing

<location>
|---02_Flames
|   |---camera_0001.mp4
|   |---camera_0002.mp4
|---09_Alexa

2. Our codebase expects the following directory structure for immersive dataset (raw video, decoded images, distorted and undistorted) after preprocessing:

<location>
|---02_Flames
|   |---camera_0001
|   |---camera_0001.mp4
|   |---camera_<...>
|---02_Flames_dist
|   |---colmap_<0>
|   |---colmap_<...>
|   |---colmap_<299>
|---02_Flames_undist
|   |---colmap_<0>
|   |---colmap_<...>
|   |---colmap_<299>
|---09_Alexa
|---09_Alexa_dist
|---09_Alexa_undist

3. Copy the picked views files to the scene dir. The views is generated by inferencing our model initialized with duration=1 points without training. We provide generated views in pkl for reproducity and simplicity.

• For example, for the scene 09_Alexa with distortion model. copy configs/im_view/09_Alexa/pickview.pkl to <location>/09_Alexa_dist/pickview.pkl

Training

You can train our model by running the following command: </br>

conda activate feature_splatting
python train.py --quiet --eval --config configs/<dataset>_<lite|full>/<scene>.json --model_path <path to save model> --source_path <location>/<scene>/colmap_0

In the argument to --config, <dataset> can be n3d (for Neural 3D Dataset) or techni (for Technicolor Dataset), and you can choose between full model or lite model. </br> You need 24GB GPU memory to train on the Neural 3D Dataset. </br> You need 48GB GPU memory to train on the Technicolor Dataset. </br> The large memory requirement is because training images are loaded into GPU memory. </br>

• For example, if you want to train the lite model on the first 50 frames of the cook_spinach scene in the Neural 3D Dataset, you can run the following command </br>

python train.py --quiet --eval --config configs/n3d_lite/cook_spinach.json --model_path log/cook_spinach_lite --source_path <location>/cook_spinach/colmap_0 

• If you want to train the full model, you can run the following command </br>

python train.py --quiet --eval --config configs/n3d_full/cook_spinach.json --model_path log/cook_spinach/colmap_0 --source_path <location>/cook_spinach/colmap_0 

Please refer to the .json config files for more options.

• If you want to train the full model with distorted immersive dataset, you can run the following command </br>

PYTHONDONTWRITEBYTECODE=1 python train_imdist.py --quiet --eval --config configs/im_distort_full/02_Flames.json --model_path log/02_Flames/colmap_0 --source_path <location>/02_Flames_dist/colmap_0 

Note, sometimes pycache file somehow affects the results. Please remove every pycache file and retrain the model without generating BYTECODE by PYTHONDONTWRITEBYTECODE=1.

• If you want to train the lite model with undistorted immersive dataset.
  Note, we remove the --eval to reuse the loader of technicolor and also to train with all cameras. gtmask 1 is specially for training with undistorted fisheye images that have black pixels.

python train.py --quiet --gtmask 1 --config configs/im_undistort_lite/02_Flames.json --model_path log/02_Flames/colmap_0 --source_path <location>/02_Flames_undist/colmap_0 

Please refer to the .json config files for more options.

Testing

• Test model on Neural 3D Dataset

python test.py --quiet --eval --skip_train --valloader colmapvalid --configpath configs/n3d_<lite|full>/<scene>.json --model_path <path to model> --source_path <location>/<scene>/colmap_0

• Test model on Technicolor Dataset

python test.py --quiet --eval --skip_train --valloader technicolorvalid --configpath configs/techni_<lite|full>/<scene>.json --model_path <path to model> --source_path <location>/<scenename>/colmap_0

• Test on Google Immersive Dataset with distortion camera model

Fist Install fused mlp layer.

pip install thirdparty/gaussian_splatting/submodules/forward_full

PYTHONDONTWRITEBYTECODE=1 CUDA_VISIBLE_DEVICES=0 python test.py --quiet --eval --skip_train --valloader immersivevalidss --configpath configs/im_distort_<lite|full>/<scene>.json --model_path <path to model> --source_path <location>/<scenename>/colmap_0

Real-Time Viewer

The viewer is based on SIBR and [Gaussian Splatting](h