QuestRoomScan

Real-time 3D room reconstruction on Meta Quest 3. Produces a textured mesh from depth + RGB camera data using GPU TSDF volume integration and Surface Nets mesh extraction, with server-based Gaussian Splat training and on-device rendering via Unity Gaussian Splatting.

| Scanning (Triplanar) | Vertex Colors | |:---:|:---:| | | |

Full demo video — recorded before multi-view blending, GPU sharpening, and atlas enhancement were added; current output quality is noticeably better.

If you're integrating this package into a game or app, start with Installation, Quick Start, and Game Integration Guide. For the full scan pipeline and debug tooling, see Usage Flow and VR Debug Menu.

Features
Requirements
Installation
Quick Start
Usage Flow
Gaussian Splat Pipeline
VR Debug Menu
Memory Budget (Quest 3)
Comparison with Hyperscape
Game Integration Guide
Credits & Prior Art
License

Features

GPU TSDF Integration — Depth frames fused into a signed distance field via compute shaders
GPU Surface Nets Meshing — Fully GPU-driven mesh extraction via compute shaders with zero CPU readback, rendered via a single Graphics.RenderPrimitivesIndirect draw call
Two-Layer Real-Time Texturing — Triplanar world-space cache (~8mm/texel persistent surface color from passthrough RGB) with vertex color fallback (~5cm). Triplanar can be disabled via inspector toggle to save ~192MB GPU memory when not needed (e.g., if only post-scan refined textures matter). Keyframes captured as motion-gated JPEGs to disk for texture refinement and Gaussian Splat training.
Package-Based Persistence — Multi-scan persistence system where each scan is a self-contained package (pkg_YYYYMMDD_HHMMSS/) with its own TSDF, triplanar textures, keyframes, splat, and refined textures. Scan browser in the debug menu lists all saved packages. Artifacts (splat, refined, HQ) auto-save to the active package on creation.
OVRSpatialAnchor Relocation — RoomAnchorManager creates a persisted OVRSpatialAnchor per scan package for reliable cross-session relocation. Per-artifact creation matrices in anchor.json track when each artifact was created relative to the spatial anchor, enabling accurate relocation even for artifacts created across different sessions. Falls back to MRUK floor anchor if spatial anchor localization fails.
Temporal Stabilization — Adaptive per-vertex temporal blending on GPU prevents mesh jitter while allowing fast convergence
Exclusion Zones — Cylindrical rejection around tracked heads prevents body reconstruction (configurable radius and height, up to 64 zones)
Gaussian Splat Training & Rendering — Keyframe capture + point cloud export → PC server training → trained PLY download → on-device UGS rendering
VR Debug Menu — Two-panel world-space UI Toolkit HUD with left navigation (Scan, Saved Scans, Refine, Gaussian Splat, Tools) and right detail views. Includes scan browser with load/delete per package (with delete confirmation), "Load Refined Only" for fast game-mode loading, context-sensitive artifact deletion, and dynamic button disabled states. Scene Objects toggle with live count. Navigation tabs for Refine and Gaussian Splat are automatically disabled when their respective modules are not attached.
Texture Refinement — Post-scan texture refinement using captured keyframes. GPU compute shader bakes a UV atlas from the best-scoring keyframe projections per texel, with multi-view blending, occlusion-aware depth testing, GPU unsharp-mask sharpening, and Sobel normal map generation for real-time lighting. Produces sharp, seamless textures with surface detail from captured keyframes. TextureRefinement is an instance-based MonoBehaviour module — all configuration (xatlas options, bake settings, sharpen/seam parameters) is via inspector fields on the component.
Atlas Enhancement (HQ Refine) — Server-side atlas super-resolution via Real-ESRGAN (2x/4x configurable) + LaMa inpainting. Uploads the on-device refined atlas as PNG, enhances, and downloads the result. Configurable SR scale via inspector.
Mesh Enhancement — Server-side mesh smoothing via bilateral normal filter + optional RANSAC plane detection and vertex snapping. Enhanced mesh saved as a separate artifact preserving the original refined mesh.
Render Mode Switching — Cycle between Wireframe, Vertex, Triplanar, Refined, Occlusion, Splat, and None at runtime via debug menu or controller binding (default: A/X button). Unavailable modes are automatically skipped during cycling (e.g., Triplanar requires TriplanarCache, Occlusion/Refined require refinement, Splat requires trained data).
Freeze Tint Toggle — Independent toggle (not tied to render mode) shows/hides a blue tint overlay on frozen voxels in live mesh modes (Vertex, Triplanar, Wireframe). Bindable via RoomScanInputHandler.
Game Integration APIs — RoomScanSession provides a high-level facade: StartScan() → await FinalizeScanAsync() → ScanResult with mesh + atlas. LoadLatestAsync() for instant game-mode loading. For finer control: LoadRefinedOnlyAsync() (loads only refined mesh + atlas, no TSDF, < 1 second), ReleaseScanResources(), public RefinedMesh/RefinedAtlas properties, RefinedMeshReady event, and ScanCoverage/ScanProgress metrics for guided UX. Scene understanding accessible via SceneObjectRegistry for MRUK + AI detected objects.
Post-Bake Mesh Simplification — UV-preserving mesh simplification via meshopt_simplifyWithAttributes runs after atlas baking (configurable ratio), preserving texture quality. Replaces the old broken pre-bake decimation.
AI Object Detection — Optional YOLO-based object detection via Unity Inference Engine (Sentis) running during scanning. GPU Non-Maximum Suppression via compute shader (only ~500 bytes readback vs ~200KB for CPU NMS). Detected objects projected to 3D world space via GPU depth projection with temporal snapshot to handle async inference. Head angular velocity gating skips blurry frames. Detection keyframes saved with JSONL metadata for post-processing.
MRUK Scene Understanding — RoomUnderstanding module populates a SceneObjectRegistry from Meta's Mixed Reality Utility Kit anchors (walls, floor, ceiling, bed, TV, doors, windows, furniture). Uses SceneModel.V2FallbackV1 with high-fidelity scene mesh for reliable detection. Event-driven anchor updates.
Scene Object Debug Visualization — Toggle world-space wireframe bounding boxes + billboard labels for all detected objects (MRUK + AI). Rendered via DebugOverlay.shader with per-source color coding (cyan = MRUK, yellow = AI). Count shown in debug menu button.
Sobel Normal Maps — GPU Sobel edge detection in AtlasBakeCompute.compute generates normal maps from the baked atlas. RefinedMesh.shader uses Sobel normals for real-time lighting on the refined mesh, adding depth and surface detail.

Requirements

Unity 6 (6000.x)
URP (Universal Render Pipeline)
Meta Quest 3 (depth sensor required)

Dependencies

| Package | Version | Notes | |---------|---------|-------| | com.unity.xr.arfoundation | 6.1+ | Depth frame access | | com.unity.render-pipelines.universal | 17.0+ | URP rendering pipeline | | com.meta.xr.mrutilitykit | 85+ | Passthrough camera RGB access | | com.unity.burst | 1.8+ | Required by Collections/Mathematics | | com.unity.collections | 2.4+ | NativeArray for plane detection | | com.unity.mathematics | 1.3+ | Math types used throughout | | org.nesnausk.gaussian-splatting | fork | Optional — Gaussian splat rendering with runtime PLY loading | | com.unity.ai.inference | 2.x+ | Optional — AI object detection (YOLO via Sentis). Assembly Genesis.RoomScan.AIDetection auto-activates when present |

Additional project-level dependencies (not in package.json — installed via Meta's SDK or XR plugin management):

com.unity.xr.meta-openxr (bridges Meta depth to AR Foundation)
com.unity.xr.openxr (OpenXR runtime)
com.meta.xr.sdk.core (OVRInput, OVRCameraRig)

Android Permissions

com.oculus.permission.USE_SCENE (depth API / spatial data)
horizonos.permission.HEADSET_CAMERA (passthrough camera RGB access)

Installation

Add to your project's Packages/manifest.json:

{
  "dependencies": {
    "com.genesis.roomscan": "https://github.com/arghyasur1991/QuestRoomScan.git"
  }
}

For Gaussian Splat support, also add the optional dependency:

{
  "dependencies": {
    "com.genesis.roomscan": "https://github.com/arghyasur1991/QuestRoomScan.git",
    "org.nesnausk.gaussian-splatting": "https://github.com/arghyasur1991/UnityGaussianSplatting.git?path=package#main"
  }
}

For AI object detection (YOLO), add the Sentis inference package:

{
  "dependencies": {
    "com.genesis.roomscan": "https://git

QuestRoomScan

Install / Use

README

QuestRoomScan

Table of Contents

Features

Requirements

Dependencies

Android Permissions

Installation