Neurecon

• Volume rendering + 3D implicit surface
• Showcase
• What?
  • previous: surface rendering; now: volume rendering
  • previous: NeRF's volume density; now: implicit surface
  • What's different of the implemented papers:
  • Future
• Results and trained models
• USAGE
• NOTES
• TODO
• CITATION
• Contact
• 🎉🎉🎉 We are hiring!

Volume rendering + 3D implicit surface = Neural 3D Reconstruction

Multi-view 3D reconstruction using neural rendering.

This repository holds :warning:unofficial:warning: pytorch implementations of:

• Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction

  • [preprint]

• NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction

  • [preprint] [official implementation]

• [VolSDF] Volume rendering of neural implicit surfaces

  • [preprint]

and more...

Showcase

Trained with VolSDF@200k, with NeRF++ as background.

• Above: :rocket: <u>volume rendering</u> of the scene (<u>novel view synthesis</u>)

• Below: mesh extracted from the learned implicit shape

| | | :----------------------------------------------------------: | | full-res video: (35 MiB, 15s@576x768@30fps) [click here] |

Trained with NeuS @300k, with NeRF++ as background.

• Above: :rocket: <u>volume rendering</u> of the scene (<u>novel view synthesis</u>)
• Middle: normals extracted from the learned implicit shape ($\nabla_{\mathbf{x}} s$)
• Below: mesh extracted from the learned implicit shape

| | | | | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | | | | |

What?

The overall topic of the implemented papers is multi-view surface and appearance reconstruction from pure posed images.

• studying and bridging between [DeepSDF/OccupancyNet]-like implicit 3D surfaces and volume rendering (NeRF).
• framework:

| What's known / Ground Truth / Supervision | What's learned | | ------------------------------------------------------------ | ----------------------------------- | | ONLY Multi-view posed RGB images. (no masks, no depths, no GT mesh or pointclouds, nothing.) | 3D surface / shape<br>3D appearance |

previous: surface rendering; now: volume rendering

From one perspective, the implemented papers introduce volume rendering to 3D implicit surfaces to differentiably render views and reconstructing scenes using photometric reconstruction loss.

| Rendering methods in previous surface reconstruction approach | Rendering method in this repo (when training) | | ------------------------------------------------------------ | --------------------------------------------- | | Surface rendering | Volume rendering |

The benefit of using volume rendering is that it diffuses gradients widely in space, and can efficiently learns a roughly correct shape at the very early beginning of training without mask supervision, avoiding bad local minimas when learning shapes, which is often encountered when using surface rendering even with mask supervision.

| config:<br> [click me] | @0 iters | @3k iters<br>@16 mins | @10k iters<br> @1 hours | @200k iters<br>@ 18.5 hours | | ------------------------------------------------------------ | ------------------------------- | ------------------------------- | ------------------------------- | --------------------------------- | | Mesh extracted from the learned shape | | | | | | View rendered from the learned appearance | | | | |

previous: NeRF's volume density; now: implicit surface

From another perspective, they change the original NeRF's shape representation (volume density $\sigma$) to a 3D implicit surface model, whose iso-surface is defined to represent spatial surfaces.

| Shape representation in NeRF | Shape representation in this repo | | ---------------------------- | ------------------------------------------------------------ | | Volume density | Occupancy net (UNISURF) <br>DeepSDF (VolSDF/NeuS) |

The biggest disadvantage of NeRF's shape representation is that it considers objects as volume clouds, which actually does not guarantees an exact surface, since there is no constraint on the learned density.

Representing shapes with implicit surfaces can force the volume density to be associated with a exact surface.

What's more, the association (or, the mapping function that maps implicit surface value to volume density) can be controlled either manually or by learn-able parameters, allowing the shape representation to be more surface-like or more volume-like, meeting different needs of different training stages.

| <img src="media/sdf2sigma.gif" alt="sdf2sigma" style="zoom:50%;" /> | | :----------------------------------------------------------: | | Demonstration of controllable mappings from sdf value to volume density value. @VolSDF |

Hence, the training scheme of approaches in this repo can be roughly divided as follows (not discrete stages, continuously progressing instead):

• at the earlier stage of learning shapes, the shape representation is more volume-like, taking into account more neighboring points along the ray when rendering colors. The network fast learns a roughly correct shape and appearance.
• while in the later stage, the shape representation is more surface-like, almost only taking into account the exact intersected point with the surface along the ray . The network slowly learns the fine thin structures of shapes and fine details of appearance.

You can see that as the controlling parameter let narrower and narrower neighboring points being considered during volume rendering, the rendered results are getting almost equivalent to surface rendering. This is proved in UNISURF, and also proved with results showed in the section [docs/usage.md#use surface rendering instead of volume rendering].

What's different of the implemented papers:

• how to map a implicit surface value to a volume density representation, or how to (accurately) calculate volume rendering's opacity with such exact surface representation.
• how to efficiently sample points on camera rays taking advantage of the exact surface
• You can find out more on my [personal notes] (In Chinese only).

Future

Currently, the biggest problem of methods contained in this repo is that the view-dependent reflection effect is baked into the object's surface, similar with IDR, NeRF and so on. In other words, if you place the learned object into a new scene with different ambient light settings, the rendering process will have no consideration of the new scene's light condition, and keeps the ambient light's reflection of the old trained scene with it.

However, as the combination of implicit surface with NeRF has come true, ambient light and material decomposition can be much easier for NeRF-based frameworks, since now shapes are represented by the underlying neural surface instead of volume densities.

Results and trained models

The trained models are stored in [[GoogleDrive]](https://dr