3D Gaussian splatting for Three.js

Three.js-based implemetation of a renderer for 3D Gaussian Splatting for Real-Time Radiance Field Rendering, a technique for generating 3D scenes from 2D images. Their project is CUDA-based and needs to run natively on your machine, but I wanted to build a viewer that was accessible via the web.

The 3D scenes are stored in a format similar to point clouds and can be viewed, navigated, and interacted with in real-time. This renderer will work with the .ply files generated by the INRIA project, standard .splat files, or my own custom .ksplat files, which are a trimmed-down and compressed version of the original .ply files.

When I started, web-based viewers were already available -- A WebGL-based viewer from antimatter15 and a WebGPU viewer from cvlab-epfl -- However no Three.js version existed. I used those versions as a starting point for my initial implementation, but as of now this project contains all my own code.

:warning: As many of you have noticed, this repo is no longer in active development. It was more of a side project and I don't currently have anywhere near enough bandwidth to devote meaningful development time to it. I recommend checking out Spark as an alternative -- it is very much in active development by the talented folks at World Labs and the open-source community, and contains a bunch of advanced features and capabilities. <br> <br>

Highlights

• Rendering is done entirely through Three.js
• Code is organized into modern ES modules
• Built-in viewer is self-contained so very little code is necessary to load and view a scene
• Viewer can import .ply files, .splat files, or my custom compressed .ksplat files
• Users can convert .ply or .splat files to the .ksplat file format
• Allows a Three.js scene or object group to be rendered along with the splats
• Built-in WebXR support
• Supports 1st and 2nd degree spherical harmonics for view-dependent effects
• Focus on optimization:
  • Splats culled prior to sorting & rendering using a custom octree
  • WASM splat sort: Implemented in C++ using WASM SIMD instructions
  • Partially GPU accelerated splat sort: Uses transform feedback to pre-calculate splat distances

Tips

• Progressively loaded .ply and .splat files will not have certain optimizations such as cache-optimized splat ordering applied to them. For optimial performance, convert these file types to .ksplat or load them non-progressively.
• Converting your scenes to .ksplat will result in the fastest loading times since its format matches the internal format for splat data.
• Scenes with large dimensions or high splat density will cause issues with the default settings. For those scenes, you can try a couple of things:
  • Set the viewer parameter integerBasedSort to false to force a slower, floating-point based splat sort.
  • Experiment with a larger value for viewer parameter splatSortDistanceMapPrecision, to adjust the precision for the distance map in the splat sort. Larger precision values will result in reduced performance, but often can alleviate visual artifacts that arise when the precision is too low.

Known issues

• Splat sort runs on the CPU – would be great to figure out a GPU-based approach
• Artifacts are visible when you move or rotate too fast (due to CPU-based splat sort)
• Sub-optimal performance on mobile devices
• Custom .ksplat file format still needs work, especially around compression
• Scenes with very large dimensions will probably crash (often with an Index out of bounds error from the splat sort). Changing splatSortDistanceMapPrecision or integerBasedSort will probably not help in those cases.

Limitations

Currently there are limits on the number of splats that can be rendered, and those limits depend mainly on the degree of spherical harmonics desired. Those limits are:

| Spherical harmonics degree | Max splat count | --- | --- | 0 | ~ 16,000,000 | 1 | ~ 11,000,000 | 2 | ~ 8,000,000

Future work will include optimizing how splat data is packed into data textures, which will help increase these limits.

Future work

This is still very much a work in progress! There are several things that still need to be done:

• Improve the way splat data is packed into data textures
• Continue optimizing CPU-based splat sort - maybe try an incremental sort of some kind?
• Support very large scenes (streaming sections & LOD)

Online demo

https://projects.markkellogg.org/threejs/demo_gaussian_splats_3d.php

Controls

Mouse

• Left click to set the focal point
• Left click and drag to orbit around the focal point
• Right click and drag to pan the camera and focal point

Keyboard

• C Toggles the mesh cursor, showing the intersection point of a mouse-projected ray and the splat mesh

• I Toggles an info panel that displays debugging info:

  • Camera position
  • Camera focal point/look-at point
  • Camera up vector
  • Mesh cursor position
  • Current FPS
  • Renderer window size
  • Ratio of rendered splats to total splats
  • Last splat sort duration

• U Toggles a debug object that shows the orientation of the camera controls. It includes a green arrow representing the camera's orbital axis and a white square representing the plane at which the camera's elevation angle is 0.

• Left arrow Rotate the camera's up vector counter-clockwise

• Right arrow Rotate the camera's up vector clockwise

• P Toggle point-cloud mode, where each splat is rendered as a filled circle

• = Increase splat scale

• - Decrease splat scale

• O Toggle orthographic mode

<br>

Building from source and running locally

Navigate to the code directory and run

npm install

Next run the build. For Linux & Mac OS systems run:

npm run build

For Windows I have added a Windows-compatible version of the build command:

npm run build-windows

To view the demo scenes locally run

npm run demo

The demo will be accessible locally at http://127.0.0.1:8080/index.html. You will need to download the data for the demo scenes and extract them into

<code directory>/build/demo/assets/data

The demo scene data is available here: https://projects.markkellogg.org/downloads/gaussian_splat_data.zip <br> <br>

Installing as an NPM package

If you don't want to build the library from source, it is also available as an NPM package. The NPM package does not come with the source code or demos that are available in the source repository. To install, run the following command:

npm install @mkkellogg/gaussian-splats-3d

<br>

Basic Usage

To run the built-in viewer:

import * as GaussianSplats3D from '@mkkellogg/gaussian-splats-3d';

const viewer = new GaussianSplats3D.Viewer({
    'cameraUp': [0, -1, -0.6],
    'initialCameraPosition': [-1, -4, 6],
    'initialCameraLookAt': [0, 4, 0]
});
viewer.addSplatScene('<path to .ply, .ksplat, or .splat file>', {
    'splatAlphaRemovalThreshold': 5,
    'showLoadingUI': true,
    'position': [0, 1, 0],
    'rotation': [0, 0, 0, 1],
    'scale': [1.5, 1.5, 1.5]
})
.then(() => {
    viewer.start();
});

Viewer parameters <br>

| Parameter | Purpose | --- | --- | cameraUp | The natural 'up' vector for viewing the scene (only has an effect when used with orbit controls and when the viewer uses its own camera). Serves as the axis around which the camera will orbit, and is used to determine the scene's orientation relative to the camera. | initialCameraPosition | The camera's initial position (only used when the viewer uses its own camera). | initialCameraLookAt | The initial focal point of the camera and center of the camera's orbit (only used when the viewer uses its own camera). <br>

Parameters for addSplatScene() <br>

| Parameter | Purpose | --- | --- | format | Force the loader to assume the specified file format when loading a splat scene. This is useful when loading from a URL where there is no file extension. Valid values are defined in the SceneFormat enum: Ply, Splat, and KSplat. | splatAlphaRemovalThreshold | Tells addSplatScene() to ignore any splats with an alpha less than the specified value (valid range: 0 - 255). Defaults to 1. | showLoadingUI | Displays a loading spinner and/or loading progress bar while the scene is loading. Defaults to true. | position | Position of the scene, acts as an offset from its default position. Defaults to [0, 0, 0]. | rotation | Rotation of the scene represented as a quaternion, defaults to [0, 0, 0, 1] (identity quaternion). | scale | Scene's scale, defaults to [1, 1, 1]. | progressiveLoad | Progressively load the scene's splat data and allow the scene to be rendered and viewed as the splats are loaded. Option is only valid for addSplatScene(), and not for addSplatScenes().

<br>

Viewer can also load multiple scenes simultaneously with the addSplatScenes() function:

import * as GaussianSplats3D from '@mkkellogg/gaussian-splats-3d';

viewer.addSplatScenes([{
        'path': '<path to first .ply, .ksplat, or .splat file>',
        'splatAlphaRemovalThreshold': 20
    },
    {
        'path': '<path to second .ply, .ksplat, or .splat file>',
        'rotation': [-0.14724434, -0.0761755, 0.1410657, 0.976020],
        'scale': [1.5, 1.5, 1.5],
        'position': [-3, -2, -3.2]
    }
])
.then(() => {
    viewer.start();
});

The addSplatScene() and addSplatScenes() methods will accept the original .ply files, standard .splat files, and my custom .ksplat files.

<br>