SAM3DBody-cpp

Standalone C++ inference engine for SAM-3D-Body — zero Python dependency at runtime.

Takes a BGR image and produces per-person MHR body pose parameters, camera translation, and optionally full 3D mesh vertices + 70 body/hand keypoints, all via ONNX Runtime + ggml.

Also includes Python frontends that call the compiled shared library via ctypes, and a CSV exporter for the 70 MHR keypoints.

🎬 Multi-person BVH motion-capture export

--bvh PATH writes a standard BVH motion-capture file per detected person (p_0.bvh, p_1.bvh, …). Identities are kept stable across frames by a built-in 2D-bbox IoU tracker, each file's joint OFFSETs are auto-resized to the actor's measured bone lengths, and the output drops straight into Blender / BVHTester / any DCC. A bundled blender/blender_bvh_plugin.py drives a MakeHuman-rigged character from the result. See BVH export for details.

./fast_sam_3dbody_run --from clip.mp4 --bvh ./p.bvh --headless
# → p_0.bvh, p_1.bvh, …

<p align="center"><em>If you want to see the repo in action, there is a <a href="https://www.youtube.com/watch?v=f-tCwCQvurQ">Youtube video here</a></em></p>

---

Models

Pre-built ONNX / GGUF / LBS model files are hosted on HuggingFace:

https://huggingface.co/AmmarkoV/SAM3DBody-cpp-onnx-models

With a CUDA GPU (recommended)

Download the all-in-one zip, extract it, and place the resulting onnx/ directory at the repo root:

wget https://huggingface.co/AmmarkoV/SAM3DBody-cpp-onnx-models/resolve/main/SAM3DBody-cpp-onnx-models.zip
unzip SAM3DBody-cpp-onnx-models.zip
# onnx/ is now at the repo root — ready to build

| File | Size | Description | |------|------|-------------| | onnx/backbone.onnx + .data | ~4.8 GB | DINOv3-ViT-H/14+ encoder (BF16, CUDA EP only) | | onnx/decoder.onnx | ~93 MB | 6-layer PromptableDecoder | | onnx/yolo.onnx | ~81 MB | YOLO11m-pose person detector | | onnx/pipeline.gguf | ~5 MB | MHR + camera projection heads | | onnx/body_model.lbs | ~27 MB | Native C LBS data (joints, weights, shape) | | onnx/correctives.bin | ~33 MB | Pose corrective blend shapes | | onnx/keypoint_mapping.bin | ~8 KB | MHR-70 keypoint index map |

Without a CUDA GPU (CPU-only)

The standard backbone.onnx and decoder.onnx are both exported in BFloat16 and require a CUDA GPU — the ORT CPU execution provider has no BF16 kernels, so it refuses to load either one (Could not find an implementation for Expand(13) … / MatMul(13) …). CPU-runnable replacements for both are separate downloads from the same HuggingFace repo.

Download these four files and place them alongside the rest of the models in onnx/:

| File | Size | Description | Links | |------|------|-------------| ---- | | backbone_fp32.onnx | ~1 MB | Graph (references external data) | Link | | backbone_fp32.onnx.data | ~3.2 GB | Float32 weights — no BF16, CPU EP compatible | Link | | decoder_fp16.onnx | ~275 KB | Graph (references external data) | Link | | decoder_fp16.onnx.data | ~97 MB | Float16 weights — no BF16, CPU EP compatible | Link |

Once they are in onnx/, --cuda -1 picks both up by itself — no extra flags:

./build/fast_sam_3dbody_run \
    --onnx-dir ./onnx \
    --cuda     -1 \
    --from     your_video.mp4

Performance expectations for CPU inference: DINOv3-ViT-H has ~630 M parameters. On a modern laptop CPU, one backbone forward pass takes 5–15 seconds, making video processing impractical. Single-image or low-frequency use cases are feasible. For anything approaching real-time, a CUDA-capable GPU is required.

WSL2 users: if nvidia-smi works inside WSL, install the CUDA toolkit for WSL2 (do not install the full Linux driver — only the WSL2 toolkit) and use the standard backbone.onnx instead. See WSL.md for a full step-by-step WSL2 setup guide.

CMake will warn at configure time if neither onnx/ nor the zip is found.

---

Pipeline

This is not 2D-to-3D lifting. The network directly regresses 3D body model parameters from image features — no depth sensor, no floor plane, no stereo.

Starting from a raw image the pipeline does:

1. YOLO detects person bounding boxes.
2. Each crop is fed to a DINOv2-ViT-H backbone producing a [1280, 32, 32] spatial feature map.
3. A transformer decoder (conditioned on the crop's ray directions and focal length) compresses that feature map into a 1024-dim pose token.
4. Two small FFN heads (run on CPU via ggml) decode the token into:
   • 519 pose parameters — global orientation (6D continuous rotation), per-joint Euler angles for 127 joints, SMPL-like shape betas (45), hand pose (108), and face expression (72).
   • 3 camera parameters [scale, tx, ty] → world-space translation [tx, ty, tz].
5. Those parameters drive linear blend skinning (LBS) over 18 439 vertices to produce the full body mesh and 70 keypoints.

The focal length is estimated from the image diagonal and baked into the decoder's conditioning input, so the network learns to associate apparent body size in the crop with metric depth — the same monocular depth-from-body-proportions approach used by SMPL/HMR-family models. The body is placed in camera space via the predicted root translation; floor-plane recovery is a downstream step if needed.

BGR image
  │
  ▼  yolo.onnx            ONNX Runtime (CUDA EP)   person bboxes + 17 COCO keypoints
  │
  ▼  backbone.onnx        ONNX Runtime (CUDA EP)   feature map  [B, 1280, 32, 32]
  │   DINOv3-ViT-H/14+
  │
  ▼  decoder.onnx         ONNX Runtime (CUDA EP)   pose token   [B, 1024]
  │   6-layer PromptableDecoder
  │
  ▼  pipeline.gguf        CPU matmul (ggml)         MHR params [B, 519] + camera [B, 3]
  │   MHR head + camera head weights
  │
  ▼  body_model.lbs       native C LBS (optional)   vertices [18439, 3] in metres
      extracted once by tools/extract_lbs_data.py

Note: body_model.onnx export is blocked on PyTorch ≥ 2.x (torch.export rejects TorchScript modules). The native C LBS path reads body_model.lbs directly and produces identical output to Python mhr_forward (body model stores data in cm; mhr_lbs_compute applies ×0.01 to match Python's /100 conversion).

Per-person output (MHRResult / FsbResult):

| Field | Shape | Description | |-------|-------|-------------| | bbox | [4] | x1 y1 x2 y2 in original image pixels | | focal_length | scalar | Estimated focal length (pixels) | | pred_cam_t | [3] | Raw camera head output: [s, tx, ty] | | global_rot | [3] | Global orientation – Euler ZYX (radians) | | body_pose | [133] | Body joint angles – Euler | | shape | [45] | SMPL-like identity blend shape betas | | scale | [28] | Scale PCA components | | hand_pose | [108] | Hand joints: left [54] + right [54] | | face_params | [72] | Facial expression parameters | | mhr_model_params | [204] | Assembled LBS parameter vector (passed to mhr_lbs_compute) | | yolo_kps | [51] | COCO 17 keypoints × [x, y, confidence] | | pred_vertices | [55317] | 18439 verts × 3, metres (when native C LBS runs) | | kps_3d | [210] | 70 joints × 3, metres (when native C LBS runs) | | kps_2d | [140] | 70 joints × 2 projected (when native C LBS runs) |

📄 See OUTPUT.md for the full reference on every 2D/3D point set and its labels — the COCO-17, MHR-70 and MHR-127 keypoint name tables, coordinate spaces, and the .joints / .obj / .csv / .bvh file formats.

---

Directory layout

SAM3DBody-cpp/
├── CMakeLists.txt
├── body_mesh.tri                     SMPL-like body mesh for the GL renderer
├── fast_sam_3dbody_frontend.py       Python lightweight frontend (ctypes, no extra deps)
├── fast_sam_3dbody_frontend-3D.py    Python 3D frontend (ctypes + Python body model)
├── fast_sam_3dbody_dump_csv.py       Python CSV exporter – 70 MHR keypoints per frame
├── two_pass.py                       Second-pass temporal smoother
├── ros_demo_webcam.py                ROS demo
├── onnx/                             Runtime model files – download from HuggingFace (see above)
│   ├── backbone.onnx + .data         ~4.8 GB  DINOv3-ViT-H/14+ encoder
│   ├── decoder.onnx                  ~93 MB   6-layer PromptableDecoder
│   ├── pipeline.gguf                 ~5 MB    MHR + camera heads
│   ├── yolo.onnx                     ~81 MB   YOLO11m-pose
│   ├── body_model.lbs                ~27 MB   native C LBS data
│   ├── correctives.bin               ~33 MB   pose corrective blend shapes
│   └── keypoint_mapping.bin          ~8 KB    MHR-70 keypoint index map
├── GraphicsEngine/
│   ├── System/glx3.{h,c}            GLX window management
│   └── ModelLoader/                  .tri mesh loader + LBS joint transform
├── AmMatrix/                         Lightweight C matrix / quaternion library
├── render/
│   ├── fast_sam_3dbody_render.cpp    OpenGL mesh overlay renderer
│   └── mhr_pose_driver.h             LBS driver (camera matrices, vertex update)
├── scripts/
│   └── build.sh / setup.sh / webcam.sh / video.sh / offline_video.sh
└── src/
    ├── fast_sam_3dbody.h             C++ public API
    ├── fast_sam_3dbody.cpp           Pipeline implementation
    ├── fast_sam_3dbody_capi.h        Plain C API (for ctypes)
    ├── fast_sam