<img src="media/logo_blue.png" width=100> neuralsim

3D surface reconstruction and simulation based on 3D neural rendering.

This repository primarily addresses two topics:

• Efficient and detailed reconstruction of implicit surfaces across different scenarios.
  • Including object-centric / street-view, indoor / outdoor, large-scale (WIP) and multi-object (WIP) datasets.
  • Highlighted implementations include neus_in_10_minutes, neus_in_10_minutes#indoor and streetsurf.
• Multi-object implicit surface reconstruction, manipulation, and multi-modal sensor simulation.
  • With particular focus on autonomous driving datasets.

TOC

• Implicit surface is all you need !
• Ecosystem
  • Highlighted implementations
• Highlights
  • :hammer_and_wrench: Multi-object volume rendering
  • :bank: Editable assetbank
  • :camera: Multi-modal sensor simulation
• Usage
  • Installation
  • code_single Single scene
  • code_multi Multi-object scene
• Roadmap & TODOs
• Acknowledgements & citations

Implicit surface is all you need !

Single-object / multi-object / indoor / outdoor / large-scale surface reconstruction and multi-modal sensor simulation | | | | ------------------------------------------------------------ | ------------------------------------------------------------ | | :rocket: Object surface reconstruction in minutes !<br />Input: posed images <u>without mask</u><br />Get started: neus_in_10_minutes<br />Credits: Jianfei Guo<br /><img src="https://github.com/PJLab-ADG/neuralsim/assets/25529198/618ff7ab-d769-47c4-9c26-bc54674e0cb2" alt="teaser_training_bmvs_gundam" width="320"> | :rocket: Outdoor surface reconstruction in minutes !<br />Input: posed images <u>without mask</u><br />Get started: neus_in_10_minutes<br />Credits: Jianfei Guo<br /><img src="https://github.com/PJLab-ADG/neuralsim/assets/25529198/7c216407-0c60-47ae-b1c1-f3ba66c67cc2" alt="teaser_training_bmvs_village_house" width="320"> | | :rocket: Indoor surface reconstruction in minutes !<br />Input: posed images, monocular cues<br />Get started: neus_in_10_minutes#indoor<br />Credits: Jianfei Guo<br /><img src="https://github.com/PJLab-ADG/neuralsim/assets/25529198/a8ab59a3-c6e2-464e-aeda-0684bf18dbb6" width="320"> | :car: Categorical surface reconstruction in the wild !<br />Input: multi-instance multi-view categorical images<br />[To be released 2023.09]<br />Credits: Qiusheng Huang, Jianfei Guo, Xinyang Li<br /><img src="https://github.com/PJLab-ADG/neuralsim/assets/25529198/b557570f-0f88-4c55-b46d-a6d5cf7f4e45" width="320"> | | :motorway: Street-view surface reconstruction in 2 hours !<br />Input: posed images, monocular cues (and optional LiDAR)<br />Get started: streetsurf<br />Credits: Jianfei Guo, Nianchen Deng <br /> <video src="https://github.com/PJLab-ADG/neuralsim/assets/25529198/7ddfe845-013c-479a-81a6-066bd04cf97c"></video>(Refresh if video won't play) | :motorway: Street-view multi-modal sensor simulation ! <br />Using reconstructed asset-bank<br />Get started: streetsurf#lidarsim<br />Credits: Jianfei Guo, Xinyu Cai, Nianchen Deng <br/> <video src="https://github.com/PJLab-ADG/neuralsim/assets/25529198/fa6e37e1-48eb-434a-89e8-9b1c230ce50d"></video>(Refresh if video won't play) | | :motorway: Street-view multi-object surfaces reconstruction in hours !<br />Input: posed images, LiDAR, 3D tracklets<br />Get started: <br />Credits: Jianfei Guo, Nianchen Deng<br /><video src="https://github.com/PJLab-ADG/neuralsim/assets/25529198/e2f0b02e-86f0-4c06-85ce-58980e6bbf96"></video>(Refresh if video won't play) | :motorway: Street-view multi-dynamic-object surfaces reconstruction in hours !<br />:rocket: Support dynamic pedestrians, cyclists, etc.<br />Credits: Jianfei Guo<br /><video src="https://github.com/PJLab-ADG/neuralsim/assets/25529198/dbd7854e-a9db-48be-a23c-61e3592f8faf"></video>(Refresh if video won't play) | | :motorway: Street-view scenario editing !<br />Using reconstructed asset-bank<br/>Credits: Jianfei Guo, Nianchen Deng <video src="https://github.com/PJLab-ADG/neuralsim/assets/25529198/c130ab03-7b8a-4068-8c2f-9c5bea219e6d"></video>(Refresh if video won't play) | :motorway: Street-view light editing ... (WIP) |

Highlighted implementations

| Methods | :rocket: Get started | Official / Un-official | Notes, major difference from paper, etc. | | ------------------------------------------------------------ | --------------------- | -------------------------------------------- | ------------------------------------------------------------ | | StreetSurf | readme | Official | - LiDAR loss improved | | NeuralSim | readme | Official | - support foreground categories: vehicles, pedestrians, cyclists, etc.<br />- support arbitrary unannotated dynamic objects<br />- support decomposition of camera lens dirt, lens flares, etc. | | NeuS in minutes | readme | Un-official | - support object-centric datasets as well as <u>indoor</u> datasets<br />- fast and stable convergence without needing mask<br />- support using NGP / LoTD or MLPs as fg&bg representations<br />- large pixel batch size (4096) & pixel error maps | | NGP with LiDAR | readme | Un-official | - using Urban-NeRF's LiDAR loss |

Updates

• 2024-02-12 [v0.6.0]
  • Major overhaul on temporal logics & support timestamps interpolation mode
  • Support using EmerNeRF for non-annotated objects.
• 2023-09-23 [v0.5.2]
  • Support generative/shared permutohedral lattice for batched/dynamic/batched-dynamic objects. :fire: Significantly reduces the memory and time usage for training a multi-object scene: one street with dozens of vehicles and dozens of pedestrians can be trained in 3 hours on a single RTX3090!
• 2023-08-22 [v0.5.1] Finish StreetSurf open-source
• 2023-01-10 :rocket: Release first public post for the neuralsim system (Chinese ver.).
• 2022-11-17 [v0.4.2] Totally refactored to packed-info based volume buffers, for both single object training (named StreetSurf later) and multi-object training.
• 2022-08-08 [v0.3.0] Totally refactored to scene-graph management with frustum culling, major overhaul on speed performance
• 2022-07-04 [v0.2.0]
  • Major overhaul on street-view training (data loading, LoTD repr., sky/mask, sparsity & regularizations.)
  • Major overhaul on multi-object rendering (batched query & BufferComposeRenderer)
• 2022-02-08 [v0.1.0] Support single-object training (NeuS/NeRF) and multi-object training (fg=DIT-NeuS/Template-NeRF, bg=NeRF/NeuS/ACORN-NeuS)
• 2021-10-18 first commit

Highlights

:hammer_and_wrench: Multi-object volume rendering

Code: app/renderers/buffer_compose_renderer.py

> Scene graph structure

Code: app/resources/scenes.py app/resources/nodes.py

To streamline the organization of assets and transformations, we adopt the concept of generic scene graphs used in modern graphics engines like magnum.

Any entity that possesses a pose or position is considered a node. Certain nodes are equipped with special functionalities, such as camera operations or drawable models (i.e. renderable assets in AssetBank).

| Real-data scene graph | Real-data frustum culling | | ---------------------------------------------- | ------------------------------------------------------ | | | |

> Efficient and universal

We provide a universal implementation of multi-object volume rendering that supports any kind of methods built for vol