InterMimic

<p align="center"> <!-- <h1 align="center">InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion</h1> --> <h1 align="center">^{<img src="assets/logo.png" align="center" width=4% >} <strong>InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions</strong></h1> <p align="center"> <a href='https://sirui-xu.github.io/' target='_blank'>Sirui Xu</a>^{<img src="assets/Illinois.jpg" align="center" width=2% >}&emsp; <a href='https://hungyuling.com/' target='_blank'>Hung Yu Ling</a> ^{<img src="assets/Electronic-Arts-Logo.png" align="center" width=1.5% >}&emsp; <a href='https://yxw.web.illinois.edu/' target='_blank'>Yu-Xiong Wang</a>^{<img src="assets/Illinois.jpg" align="center" width=2% >}&emsp; <a href='https://lgui.web.illinois.edu/' target='_blank'>Liang-Yan Gui</a>^{<img src="assets/Illinois.jpg" align="center" width=2% >}&emsp; <br> ^{<img src="assets/Illinois.jpg" align="center" width=2% >}University of Illinois Urbana-Champaign, ^{<img src="assets/Electronic-Arts-Logo.png" align="center" width=1.5% >} Electronic Arts <br> <strong>CVPR 2025 Highlight 🏆</strong> </p> </p> </p> <p align="center"> <a href='https://arxiv.org/abs/2502.20390'> <img src='https://img.shields.io/badge/Arxiv-2502.20390-A42C25?style=flat&logo=arXiv&logoColor=A42C25'></a> <a href='https://arxiv.org/pdf/2502.20390.pdf'> <img src='https://img.shields.io/badge/Paper-PDF-yellow?style=flat&logo=arXiv&logoColor=yellow'></a> <a href='https://sirui-xu.github.io/InterMimic/'> <img src='https://img.shields.io/badge/Project-Page-green?style=flat&logo=Google%20chrome&logoColor=green'></a> <a href='https://youtu.be/ZJT387dvI9w'> <img src='https://img.shields.io/badge/YouTube-Video-EA3323?style=flat&logo=youtube&logoColor=EA3323'></a> <a href='https://www.bilibili.com/video/BV1nW9KYFEUX/'> <img src='https://img.shields.io/badge/Bilibili-Video-4EABE6?style=flat&logo=Bilibili&logoColor=4EABE6'></a> <a href='https://github.com/Sirui-Xu/InterMimic'> <img src='https://img.shields.io/badge/GitHub-Code-black?style=flat&logo=github&logoColor=white'></a> </p>

🏠 Overview

<div align="center"> <img src="assets/teaser.png" width="100%" alt="InterMimic teaser"/> </div>

InterMimic features one unified policy, spanning diverse full-body interactions with dynamic, heterogeneous objects—and it works out-of-the-box for both SMPL-X and Unitree G1 humanoids.

📹 Demo

<p align="center"> <img src="assets/InterMimic.gif" align="center" width=60% > </p>

🔥 News

• [2026-02-09] 🚀 Multi-GPU training support is here!
• [2025-12-17] 🚀 Isaac Gym checkpoints are compatible with IsaacLab inference?! Check out the newly released implementation.
• [2025-12-15] IsaacLab support is underway! Data replay is ready—more coming in the next release ☕️
• [2025-12-07] 🚀 Release a data conversion pipeline for bringing InterAct into simulation. The processing code is available in the InterAct repository.
• [2025-06-10] Release the instruction for the student policy inference.
• [2025-06-03] Initial release of PSI and the processed data. Next release: teacher policy inference for dynamics-aware retargeting, and student policy inference.
• [2025-05-26] It's been a while! The student policy training pipeline has been released! The PSI and other data construction pipelines will follow soon.
• [2025-04-18] Release a checkpoint with high‑fidelity physics and enhanced contact precision.
• [2025-04-11] The training code for teacher policies is live—try training your own policy!
• [2025-04-05] We're excited by the overwhelming interest in humanoid robot support and are ahead of schedule in open-sourcing our Unitree-G1 integration—starting with a small demo with support for G1 with its original three-finger dexterous hands. Join us in exploring whole-body loco-manipulation with humanoid robots!
• [2025-04-04] InterMimic has been selected as a CVPR Highlight Paper 🏆. More exciting developments are on the way!
• [2025-03-25] We’ve officially released the codebase and checkpoint for teacher policy inference demo — give it a try! ☕️

📖 Getting Started

Dependencies

Isaac Gym environment

1. Create a dedicated conda environment (Python 3.8) and install PyTorch + repo deps:

   conda create -n intermimic-gym python=3.8
   conda activate intermimic-gym
   conda install pytorch torchvision torchaudio pytorch-cuda=11.6 -c pytorch -c nvidia
   pip install -r requirement.txt
   

   (Alternatively, start from environment.yml, though it includes some optional extras.)

2. Install Isaac Gym following NVIDIA’s instructions.

3. Fix the Isaac Gym shared-library lookup when using conda by exporting:

   export LD_LIBRARY_PATH="$CONDA_PREFIX/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
   

   Do this after every conda activate intermimic-gym before launching Gym scripts; it ensures libpython3.8.so is discoverable.

Isaac Lab environment

• Install Isaac Lab separately by following the official guide and keep that environment isolated (typically via Isaac Sim’s python or the provided uv/conda env). Recommended version: Isaac Sim 5.1.0 with IsaacLab v2.3.1

• Export ISAACLAB_PATH once per shell session so our helper scripts (which source $ISAACLAB_PATH/isaaclab.sh) can locate your install:

  export ISAACLAB_PATH=/path/to/your/IsaacLab
  

• Optional: if you plan to use the --record-video flag in our replay script, install imageio (and imageio-ffmpeg for MP4 support) inside the Isaac Lab Python environment:

  $ISAACLAB_PATH/isaaclab.sh -p -m pip install --upgrade imageio imageio-ffmpeg
  

Data

• Download the dataset, unzip it, and move the extracted folder to InterAct/OMOMO_new/. This build contains minor fixes to the original release, so your results may deviate slightly from those reported in the paper.

• 🔥 We recommend processing the data using our InterAct to obtain richer HOI skills and higher-quality outputs than the original OMOMO dataset.

Data Replay

To replay the ground-truth data you now have two options:

Isaac Gym

sh isaacgym/scripts/data_replay.sh

Isaac Lab

./isaaclab/scripts/run_data_replay.sh --num-envs 8 --motion-dir InterAct/OMOMO_new

Helpful flags for the Isaac Lab demo:

• --num-envs: sets both cfg.num_envs and cfg.scene.num_envs.
• --headless: launches Isaac Sim without the viewer.
• --motion-dir: dataset directory relative to $INTERMIMIC_PATH.
• --no-playback: disables dataset playback so you can step physics manually.
• --record-video /path/to/video.mp4: captures RGB frames each step (requires imageio).
• --video-fps: frame rate for --record-video captures (defaults to 30 FPS).

Teacher Policy Training

To train a teacher policy, execute the following commands:

sh isaacgym/scripts/train_teacher.sh

A higher‑fidelity simulation enough for low-dynamic interaction (trading off some efficiency for realism):

sh isaacgym/scripts/train_teacher_new.sh

How to enable PSI

Open the training config, for example, omomo_train_new.yaml. Set

physicalBufferSize: <integer greater than 1>

Student Policy Training

Download the data from teacher's retargeting and correction, to train a student policy with distillation, execute the following commands:

sh isaacgym/scripts/train_student.sh

To train with transformer network architecture:

sh isaacgym/scripts/train_student_transformer.sh

🔥 Multi-GPU Training

For faster training with multiple GPUs, we provide multi-GPU versions of the training scripts. These scripts use torchrun to launch distributed training across all available GPUs.

Teacher Policy (Multi-GPU)

# Uses all available GPUs by default
sh isaacgym/scripts/train_teacher_multigpu.sh

# High-fidelity simulation variant
sh isaacgym/scripts/train_teacher_new_multigpu.sh

Student Policy (Multi-GPU)

# MLP-based student policy
sh isaacgym/scripts/train_student_multigpu.sh

# Transformer-based student policy
sh isaacgym/scripts/train_student_transformer_multigpu.sh

Specifying the Number of GPUs

By default, all available GPUs are used. To specify a different number:

NUM_GPUS=2 sh isaacgym/scripts/train_teacher_multigpu.sh

Training Hyperparameters for Multi-GPU

With multi-GPU training, gradients are averaged across all GPUs, so each update step effectively processes more data. You may want to adjust the following in your training config (e.g., omomo.yaml):

• mini_epochs: can be reduced
• minibatch_size: can be reduced

Teacher Policy Inference

We’ve released a checkpoint for one (out of 17) teacher policy on OMOMO, along with some sample data. To get started:

1. Download the checkpoints and place them in the current directory.

2. Then, run the following commands:

   sh isaacgym/scripts/test_teacher.sh
   

   For quantitative evaluation with metrics (execution steps, pose errors, success rate):

   sh isaacgym/scripts/eval_teache