SparseCam4D: Spatio-Temporally Consistent 4D Reconstruction from Sparse Cameras

Project page | Paper

SparseCam4D: Spatio-Temporally Consistent 4D Reconstruction from Sparse Cameras, Weihong Pan, Xiaoyu Zhang, Zhuang Zhang, Zhichao Ye, Nan Wang, Haomin Liu, Guofeng Zhang Zhejiang University, Inspatio Research CVPR 2026

This repository is the official implementation of "SparseCam4D: Spatio-Temporally Consistent 4D Reconstruction from Sparse Cameras".

Get started

Environment

This code is built upon 4d-gaussian-splatting and 4DGaussians. To set up the environment, please run:

conda env create --file environment.yml
conda activate sparse4d

Data preparation

We provide a demo dataset on HuggingFace.

The expected data layout is:

balloon1/
├── depth/
│   ├── cam01/              # per-frame depth maps for training camera cam01 (*.npy)
│   ├── cam06/              # per-frame depth maps for training camera cam06 (*.npy)
│   ├── cam10/              # per-frame depth maps for training camera cam10 (*.npy)
│   ├── cam01.mp4           # depth video visualization
│   ├── cam06.mp4
│   └── cam10.mp4
├── images/                 # all input images, named as <cam>_<time>.png
├── preprocess/
│   ├── time_0000/
│   │   ├── diffusion/      # pseudo-view images generated by ViewCrafter at t=0
│   │   └── sparse/0/       # COLMAP sparse reconstruction at t=0 (cameras.bin, points3D.ply, ...)
│   ├── time_0001/
│   │   └── diffusion/      # pseudo-view images at t=1
│   └── ...                 # time_0002 ~ time_0099, each with diffusion/
├── sfm_transforms_extend.json   # camera intrinsics + extrinsics for all views and timestamps
├── vc_roma_sfm_300.ply          # initial point cloud (SfM + RoMa dense matching)
├── transforms_train.json        # camera poses for training split
└── transforms_test.json         # camera poses for test split

Depth maps are estimated by Video Depth Anything on the training-camera videos.

Pseudo-view images under preprocess/time_*/diffusion/ are synthesized by ViewCrafter from training cameras to cover additional viewpoints at each timestamp with sparse camera poses estimated by VGGT.

Note: For full data preparation pipeline, please refer to preprocess.

Running

1. Training

Edit the source_path and model_path fields in the config file, then run:

python train.py --config configs/nvidia/balloon1.yaml

2. Pose alignment (before evaluation)

The SfM camera poses are in a different coordinate system from the original dataset poses. Before evaluating on test cameras, align the poses:

python scripts/postprocess_pose_sfm2ours.py \
    --model_path $model_path \
    --scene balloon1 \
    --anchor_cam cam01,cam06,cam10

This saves sfm_transforms_to_ours.json under $model_path/pose/.

3. Rendering and evaluation

python render.py --config configs/nvidia/balloon1.yaml --skip_train --iteration 30000

Acknowledgements

This project builds upon the following excellent works:

• 4d-gaussian-splatting — 4D Gaussian Splatting codebase
• 4DGaussians — HexPlane-based dynamic Gaussian representation
• ViewCrafter — video diffusion model for novel-view synthesis used in pseudo-view generation
• VGGT — feed-forward 3D reconstruction for camera pose estimation
• Video Depth Anything — temporally consistent monocular depth estimation

BibTex

@article{pan2026sparsecam4d,
  title={SparseCam4D: Spatio-Temporally Consistent 4D Reconstruction from Sparse Cameras},
  author={Pan, Weihong and Zhang, Xiaoyu and Zhang, Zhuang and Ye, Zhichao and Wang, Nan and Liu, Haomin and Zhang, Guofeng},
  journal={arXiv preprint arXiv:2603.26481},
  year={2026}
}