Text2HOI

Official code of Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction in CVPR 2024<br> Arxiv paper. / Project page.

Data

| Dataset | Resource | |-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------| | H2O | Download | | GRAB | Download | | ARCTIC | Download | | ARCTIC Text Description | Download | | MANO | Download | | Object Pickle File | Download |

Preprocessing GRAB object

GRAB objects have so many vertiecs. So we reduce the number of vertices to 4,000.

python preprocessing_grab_object.py

Folder Tree for Demo

data
├─ h2o
│  ├─ obj.pkl
│  └─ object
│     ├─ book
│     ├─ cappuccino
│     ├─ chips
│     ├─ cocoa
│     ├─ espresso
│     ├─ lotion
│     ├─ milk
│     └─ spray
|
├─ grab
│  ├─ obj.pkl
│  ├─ processed_object_meshes
│  └─ contact_meshes
│     ├─ airplane.ply
│     ├─ alarmclock.ply
│     ├─ apple.ply
│     ├─ banana.ply
│     ...
|
├─ arctic
│  ├─ obj.pkl
│  └─ downloads/data/meta/object_vtemplates
│     ├─ box
│     ├─ capsulemachine
│     ├─ espressomachine
│     ├─ ketchup
│     ...
│
└─ mano
   └─ mano_v1_2
      └─ models
         ├─ info.txt
         ├─ LICENSE.txt
         ├─ MANO_LEFT.pkl
         └─ MANO_RIGHT.pkl

Checkpoints

Download

checkpoints
├─ h2o
├─ grab
└─ arctic

Installation

source scripts/install.sh

Demo

source scripts/demo.sh

Train

Folder Tree for Preprocessing

data
├─ h2o
│  ├─ object
│  ├─ subject1
│  ├─ subject2
│  └─ subject3
|
├─ grab
│  ├─ contact_meshes
│  ├─ processed_object_meshes
│  ├─ s1
│  ├─ s2
│  ├─ ...
│  └─ s10
│
├─ arctic
│  └─ downloads
│     └─ data
│        ├─ raw_seqs
|        ├─ description   # (Download available in the 'Data - Arctic' section above or in the 'For Future Work' section below.)
│        └─ meta
│           └─ object_vtemplates
│
└─ mano
   └─ mano_v1_2
      └─ models
         ├─ info.txt
         ├─ LICENSE.txt
         ├─ MANO_LEFT.pkl
         └─ MANO_RIGHT.pkl

Data preparation (preprocessing)

python preprocessing.py

Alternatively, you can download the preprocessed data files, Download.

data
├─ h2o
│  ├─ balance_weights.pkl
│  ├─ data.npz
│  ├─ obj.pkl
│  ├─ text.json
│  ├─ text_count.json
│  └─ text_length.json
|
├─ grab
│  ├─ balance_weights.pkl
│  ├─ data.npz
│  ├─ obj.pkl
│  ├─ text.json
│  ├─ text_count.json
│  └─ text_length.json
│
└─ arctic
   ├─ balance_weights.pkl
   ├─ data.npz
   ├─ obj.pkl
   ├─ text.json
   ├─ text_count.json
   └─ text_length.json

Train Text2HOI

source scripts/train/train_texthom.sh

Train Refiner

source scripts/train/train_refiner.sh

For Future Work

Text description for Arctic.