LeRobot SO-ARM-100 6 DOF robotic arm manipulation with Genesis simulator and Feetech motors

There are 2 main use cases

1. sim to real, where genesis controls the physical robot
2. real to sim, where the physical robot moves will refresh the robot rendering in genesis

TOC

• Acknowledgements
• Installation
• Camera
• Calibration & Validation
• Real 2 Sim
• Path Planning
• Dataset Generation
• Policies
• Tele-operation
• Examples
• Gradio apps
• LeRobot dataset EDA

Goal

The goal is to replay an episode recorded in REAL into a SIM environment.

| Real | Sim Visual | Sim Collision | |------|------------|---------------| | <video controls src="https://github.com/user-attachments/assets/0cd6b8a6-f75c-4e72-adf0-ffdeddc1c45b"></video> | <video controls src="https://github.com/user-attachments/assets/1bc9a00e-fdda-4590-8fb7-ee414f0ef183"></video> | <video controls src="https://github.com/user-attachments/assets/0e8e0346-5ef1-475e-9eba-1374347e4f71"></video> |

Tele-operation

For live simulation, the tele-operation process should rely on a scalable asynchronous process.

<video controls src="https://github.com/user-attachments/assets/2e5d367d-ad09-40b2-8e69-7dbe024627f0"></video>

With pub/sub, workers repeatedly poll tasks from their dedicated queue, pushing their output to rerun.io database.

See Tele-operation for more details.

flowchart LR
    subgraph main [Cron]
        Cron["Infinite Loop - 30 Hz"]
    end

    subgraph resources [Resources]
        LeaderArm[Leader Arm]
        FollowerArm[Follower Arm]
        Webcam[Webcam]
        Genesis[Genesis]
        YOLO26[YOLO26]
    end

    subgraph workers [Worker Processes]
        LeaderRead[Leader Read]
        FollowerControl[Follower Control]
        WebcamCapture[Webcam Capture]
        DetectObjects[Detect Objects]
        SimStep[Sim Step]
    end

    subgraph fifos [FIFO Queues]
        Q1([leader_read_q])
        Q2([follower_control_q])
        Q3([sim_step_q])
        Q4([webcam_capture_q])
    end

    subgraph shm [SHM]
        SHM{{Shared Memory}}
    end

    subgraph metrics [Database]
        Rerun[(Rerun.io)]
    end

    Cron -->|"empty"| Q1
    Q1 --> LeaderRead
    LeaderRead -->|"qpos[]"| Q2
    Q2 --> FollowerControl
    FollowerControl -->|"empty"| Q4
    Q4 --> WebcamCapture
    FollowerControl -->|"qpos[]"| Q3
    Q3 --> SimStep

    LeaderRead <-->|"qpos[]"| LeaderArm
    FollowerControl <-->|"qpos[]"| FollowerArm
    WebcamCapture <-->|"BGR[][]"| Webcam
    SimStep <-->|"RGB[][]"| Genesis

    WebcamCapture -->|"BGR[][]"| SHM
    SHM -->|"BGR[][]"| DetectObjects
    DetectObjects <-->|"predict"| YOLO26

    Cron --> Rerun
    LeaderRead --> Rerun
    FollowerControl --> Rerun
    WebcamCapture --> Rerun
    DetectObjects -->|"BoundingBox[]"| Rerun
    SimStep --> Rerun