Roboreg

Robot registration (roboreg): eye-to-hand calibration from RGB / RGB-D images using robot mesh as calibration target.

Generate Convert Improve

Install / Use

/learn @lbr-stack/Roboreg

About this skill

Quality Score

0/100

README

roboreg

Eye-to-hand calibration from RGB-D images using robot mesh as calibration target.

<body> <table> <caption>Mesh (purple) and Point Cloud (turqoise).</caption> <tr> <th align="left" width="50%">Unregistered</th> <th align="left" width="50%">Registered</th> </tr> <tr> <td align="center"><img src="https://raw.githubusercontent.com/lbr-stack/roboreg/refs/heads/main/doc/img/hydra_robust_icp_unregistered.png" alt="Unregistered Mesh and Point Cloud"></td> <td align="center"><img src="https://raw.githubusercontent.com/lbr-stack/roboreg/refs/heads/main/doc/img/hydra_robust_icp_registered.png" alt="Registered Mesh and Point Cloud"></td> </tr> </table> </body>

Installation
- Pip (Requires CUDA Toolkit Installation)
- Docker (Comes with CUDA Toolkit)
Command Line Interface
Testing

Installation

Two install options are provided:

Pip (Requires CUDA Toolkit Installation)
Docker (Comes with CUDA Toolkit)

Pip (Requires CUDA Toolkit Installation)

[!NOTE] During runtime, CUDA Toolkit is required for the differentiable rendering. If you are planning to do differentiable rendering, see CUDA Toolkit Install Instructions.

To pip intall roboreg, simply run

pip install roboreg

Docker (Comes with CUDA Toolkit)

A sample Docker container is provided for testing purposes. First:

Install Docker, see Docker Install Instructions
Install NVIDIA Container Toolkit, see NVIDIA Container Toolkit Install Instructions

Clone this repository

git clone git@github.com:lbr-stack/roboreg.git

Build the Docker image (currently only runtime support, i.e. no rendering via compiled kernels)

cd roboreg
docker build \
    -t roboreg:latest \
    -f .docker/Dockerfile \
    --build-arg PYTORCH_VERSION=2.10.0 \
    --build-arg CUDA_VERSION=13.0 \
    .

For CUDA 12.4, use:

docker build \
    -t roboreg:latest \
    -f .docker/Dockerfile \
    --build-arg PYTORCH_VERSION=2.6.0 \
    --build-arg CUDA_VERSION=12.4 \
    .

Run container (on Linux host with NVIDIA Container Toolkit)

docker run -it \
    --gpus all \
    --network host \
    --ipc host \
    --volume /tmp/.X11-unix:/tmp/.X11-unix \
    --volume /dev/shm:/dev/shm \
    --volume /dev:/dev --privileged \
    --env DISPLAY \
    --env QT_X11_NO_MITSHM=1 \
    roboreg:latest

Command Line Interface

[!TIP] Examples use sample data under test/assets/lbr_med7_r800. Data is stored via Git Large File Storage (LFS). Data is cloned automatically when git-lfs is installed. To clone in retrospect:
sudo apt install git-lfs
git lfs fetch --all
git lfs checkout

Segment

This is a required step to generate robot masks.

rr-sam2 \
    --path test/assets/lbr_med7_r800/samples \
    --pattern "left_image_*.png" \
    --n-positive-samples 5 \
    --n-negative-samples 5 \
    --device cuda

Hydra Robust ICP

The Hydra robust ICP implements a point-to-plane ICP registration on a Lie algebra. It does not use rendering and can also be used on CPU.

rr-hydra \
    --camera-info-file test/assets/lbr_med7_r800/samples/left_camera_info.yaml \
    --path test/assets/lbr_med7_r800/samples \
    --mask-pattern mask_sam2_left_image_*.png \
    --depth-pattern depth_*.npy \
    --joint-states-pattern joint_states_*.npy \
    --urdf-path test/assets/lbr_med7_r800/description/lbr_med7_r800.urdf \
    --root-link-name lbr_link_0 \
    --end-link-name lbr_link_7 \
    --number-of-points 5000 \
    --display-results \
    --output-file HT_hydra_robust.npy

Camera Swarm

[!WARNING] On first run, nvdiffrast compiles PyTorch extensions. This might use too many resources on some systems (< 16 GB RAM). You can create an environment variable export MAX_JOBS=1 before the first run to limit concurrent compilation. Also refer to this Issue.

The camera swarm optimization can serve for finding an initial guess to Monocular Differentiable Rendering or Stereo Differentiable Rendering.

rr-cam-swarm \
    --collision-meshes \
    --n-cameras 1000 \
    --min-distance 0.5 \
    --max-distance 3.0 \
    --angle-range 3.141 \
    --w 0.7 \
    --c1 1.5 \
    --c2 1.5 \
    --max-iterations 100 \
    --display-progress \
    --urdf-path test/assets/lbr_med7_r800/description/lbr_med7_r800.urdf \
    --root-link-name lbr_link_0 \
    --end-link-name lbr_link_7 \
    --target-reduction 0.8 \
    --scale 0.1 \
    --n-samples 1 \
    --camera-info-file test/assets/lbr_med7_r800/samples/left_camera_info.yaml \
    --path test/assets/lbr_med7_r800/samples \
    --image-pattern left_image_*.png \
    --joint-states-pattern joint_states_*.npy \
    --mask-pattern mask_sam2_left_image_*.png \
    --output-file HT_cam_swarm.npy

Monocular Differentiable Rendering

[!WARNING] On first run, nvdiffrast compiles PyTorch extensions. This might use too many resources on some systems (< 16 GB RAM). You can create an environment variable export MAX_JOBS=1 before the first run to limit concurrent compilation. Also refer to this Issue.

This monocular differentiable rendering refinement requires a good initial estimate, as e.g. obtained from Hydra Robust ICP or Camera Swarm

rr-mono-dr \
    --optimizer SGD \
    --lr 0.01 \
    --max-iterations 100 \
    --display-progress \
    --urdf-path test/assets/lbr_med7_r800/description/lbr_med7_r800.urdf \
    --root-link-name lbr_link_0 \
    --end-link-name lbr_link_7 \
    --camera-info-file test/assets/lbr_med7_r800/samples/left_camera_info.yaml \
    --extrinsics-file test/assets/lbr_med7_r800/samples/HT_hydra_robust.npy \
    --path test/assets/lbr_med7_r800/samples \
    --image-pattern left_image_*.png \
    --joint-states-pattern joint_states_*.npy \
    --mask-pattern mask_sam2_left_image_*.png \
    --output-file HT_dr.npy

Stereo Differentiable Rendering

[!WARNING] On first run, nvdiffrast compiles PyTorch extensions. This might use too many resources on some systems (< 16 GB RAM). You can create an environment variable export MAX_JOBS=1 before the first run to limit concurrent compilation. Also refer to this Issue.

This stereo differentiable rendering refinement requires a good initial estimate, as e.g. obtained from Hydra Robust ICP or Camera Swarm

rr-stereo-dr \
    --optimizer SGD \
    --lr 0.01 \
    --max-iterations 100 \
    --display-progress \
    --urdf-path test/assets/lbr_med7_r800/description/lbr_med7_r800.urdf \
    --root-link-name lbr_link_0 \
    --end-link-name lbr_link_7 \
    --left-camera-info-file test/assets/lbr_med7_r800/samples/left_camera_info.yaml \
    --right-camera-info-file test/assets/lbr_med7_r800/samples/right_camera_info.yaml \
    --left-extrinsics-file test/assets/lbr_med7_r800/samples/HT_hydra_robust.npy \
    --right-extrinsics-file test/assets/lbr_med7_r800/samples/HT_right_to_left.npy \
    --path test/assets/lbr_med7_r800/samples \
    --left-image-pattern left_image_*.png \
    --right-image-pattern right_image_*.png \
    --joint-states-pattern joint_states_*.npy \
    --left-mask-pattern mask_sam2_left_image_*.png \
    --right-mask-pattern mask_sam2_right_image_*.png \
    --left-output-file HT_left_dr.npy \
    --right-output-file HT_right_dr.npy

Render Results

[!WARNING] On first run, nvdiffrast compiles PyTorch extensions. This might use too many resources on some systems (< 16 GB RAM). You can create an environment variable export MAX_JOBS=1 before the first run to limit concurrent compilation. Also refer to this Issue.

Generate renders using the obtained extrinsics:

rr-render \
    --batch-size 1 \
    --num-workers 0 \
    --urdf-path test/assets/lbr_med7_r800/description/lbr_med7_r800.urdf \
    --root-link-name lbr_link_0 \
    --end-link-name lbr_link_7 \
    --camera-info-file test/assets/lbr_med7_r800/samples/left_camera_info.yaml \
    --extrinsics-file test/assets/lbr_med7_r800/samples/HT_left_dr.npy \
    --images-path test/assets/lbr_med7_r800/samples \
    --joint-states-p

Related Skills

node-connect

350.1k

Diagnose OpenClaw node connection and pairing failures for Android, iOS, and macOS companion apps

frontend-design

109.9k

Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, or applications. Generates creative, polished code that avoids generic AI aesthetics.

openai-whisper-api

350.1k

Transcribe audio via OpenAI Audio Transcriptions API (Whisper).

qqbot-media

350.1k

QQBot 富媒体收发能力。使用 <qqmedia> 标签，系统根据文件扩展名自动识别类型（图片/语音/视频/文件）。

lbr-stack

View profile

View on GitHub

GitHub Stars101

CategoryDevelopment

Updated18d ago

Forks3

lbr-stack/roboreg

Languages

Python

Security Score

95/100

Audited on Mar 19, 2026

No findings

Roboreg

Install / Use

README

roboreg

Table of Contents

Installation

Pip (Requires CUDA Toolkit Installation)

Docker (Comes with CUDA Toolkit)

Command Line Interface

Segment

Hydra Robust ICP

Camera Swarm

Monocular Differentiable Rendering

Stereo Differentiable Rendering

Render Results

Related Skills