<div align="center"> <img src="logo.png" width="40%"> </div>

EasyMocap is an open-source toolbox for markerless human motion capture and novel view synthesis from RGB videos. In this project, we provide a lot of motion capture demos in different settings.

python

News

:tada: Our SIGGRAPH 2022 Novel View Synthesis of Human Interactions From Sparse Multi-view Videos is released! Check the documentation.
:tada: EasyMocap v0.2 is released! We support motion capture from Internet videos. Please check the Quick Start for more details.

Core features

Multiple views of a single person

This is the basic code for fitting SMPL[^loper2015]/SMPL+H[^romero2017]/SMPL-X[^pavlakos2019]/MANO[^romero2017] model to capture body+hand+face poses from multiple views.

<div align="center"> <img src="doc/feng/mv1pmf-smplx.gif" width="80%"> <br> <img src="https://raw.githubusercontent.com/chingswy/Dataset-Demo/main/EasyMocap/mv1p-dance-smpl.gif" width="80%"> <br> <sup>Videos are from ZJU-MoCap, with 23 calibrated and synchronized cameras.</sup> </div> <div align="center"> <img src="doc/feng/mano.gif" width="80%"> <br> <sup>Captured with 8 cameras.</sup> </div>

Internet video

This part is the basic code for fitting SMPL[^loper2015] with 2D keypoints estimation[^cao2018][^hrnet] and CNN initialization[^kolotouros2019].

<div align="center"> <img src="https://raw.githubusercontent.com/chingswy/Dataset-Demo/main/EasyMocap/23EfsN7vEOA%2B003170%2B003670.gif" width="80%"> <br> <sup>The raw video is from <a href="https://www.youtube.com/watch?v=23EfsN7vEOA">Youtube</a>.</sup> </div>

Internet video with a mirror

<div align="center"> <img src="https://raw.githubusercontent.com/zju3dv/Mirrored-Human/main/doc/assets/smpl-avatar.gif" width="80%"> <br> <sup>The raw video is from <a href="https://www.youtube.com/watch?v=KOCJJ27hhIE">Youtube</a>.</sup> </div>

Multiple Internet videos with a specific action (Coming soon)

<div align="center"> <img src="doc/imocap/imocap.gif" width="80%"><br/> <sup>Internet videos of Roger Federer's serving</sup> </div>

Multiple views of multiple people

<div align="center"> <img src="doc/assets/mvmp1f.gif" width="80%"><br/> <sup>Captured with 8 consumer cameras</sup> </div>

Novel view synthesis from sparse views

<div align="center"> <img src="https://raw.githubusercontent.com/chingswy/Dataset-Demo/main/EasyMocap/female-ballet.gif" width="80%"><br/> <sup>Novel view synthesis for chanllenge motion(coming soon)</sup> </div> <div align="center"> <img src="https://raw.githubusercontent.com/chingswy/Dataset-Demo/main/EasyMocap/nvs_mp_soccer1_6_rgb.gif" width="80%"><br/> <sup>Novel view synthesis for human interaction</sup> </div>

ZJU-MoCap

With our proposed method, we release two large dataset of human motion: LightStage and Mirrored-Human. See the website for more details.

If you would like to download the ZJU-Mocap dataset, please sign the agreement, and email it to Qing Shuai (s_q@zju.edu.cn) and cc Xiaowei Zhou (xwzhou@zju.edu.cn) to request the download link.

<div align="center"> <div align="center" width="40%"> <img src="doc/assets/ZJU-MoCap-lightstage.jpg" width="40%"><br/> <sup>LightStage: captured with LightStage system</sup> </div> <div align="center" width="40%"> <img src="https://raw.githubusercontent.com/chingswy/Dataset-Demo/main/EasyMocap/mirrored-human.jpg" width="40%"><br/> <sup>Mirrored-Human: collected from the Internet</sup> </div> </div>

Many works have achieved wonderful results based on our dataset:

Other features

3D Realtime visualization

Camera calibration

<div align="center"> <img src="https://raw.githubusercontent.com/chingswy/Dataset-Demo/main/EasyMocap/calib_intri.jpg" width="40%"> <img src="https://raw.githubusercontent.com/chingswy/Dataset-Demo/main/EasyMocap/calib_extri.jpg" width="40%"> <br> <sup>Calibration for intrinsic and extrinsic parameters</sup> </div>

Annotator

<div align="center"> <img src="https://raw.githubusercontent.com/chingswy/Dataset-Demo/main/EasyMocap/annot_keypoints.jpg" width="40%"> <img src="https://raw.githubusercontent.com/chingswy/Dataset-Demo/main/EasyMocap/annot_mask.jpg" width="40%"> <br> <sup>Annotator for bounding box, keypoints and mask</sup> </div>

Updates

11/03/2022: Support MultiNeuralBody.
12/25/2021: Support mediapipe keypoints detector.
08/09/2021: Add a colab demo here.
06/28/2021: The Multi-view Multi-person part is released!
06/10/2021: The real-time 3D visualization part is released!
04/11/2021: The calibration tool and the annotator are released.
04/11/2021: Mirrored-Human part is released.

Installation

See documentation for more instructions.

Acknowledgements

Here are the great works this project is built upon:

SMPL models and layer are from MPII SMPL-X model.
Some functions are borrowed from SPIN, VIBE, SMPLify-X
The method for fitting 3D skeleton and SMPL model is similar to SMPLify-X(with 3D keypoints loss), TotalCapture(without using point clouds).
We integrate some easy-to-use functions for previous great work:
- easymocap/estimator/mediapipe_wrapper.py: MediaPipe
- easymocap/estimator/SPIN : an SMPL estimator[^cao2018]
- easymocap/estimator/YOLOv4: an object detector[^kolotouros2019]
- easymocap/estimator/HRNet : a 2D human pose estimator[^bochkovskiy2020]

Contact

Please open an issue if you have any questions. We appreciate all contributions to improve our project.

Contributor

EasyMocap is built by researchers from the 3D vision group of Zhejiang University: Qing Shuai, Qi Fang, Junting Dong, Sida Peng, Di Huang, Hujun Bao, and Xiaowei Zhou.

We would like to thank Wenduo Feng, Di Huang, Yuji Chen, Hao Xu, Qing Shuai, Qi Fang, Ting Xie, Junting Dong, Sida Peng and Xiaopeng Ji who are the performers in the sample data. We would also like to thank all the people who has helped EasyMocap in any way.

Citation

This project is a part of our work iMocap, [Mirrored-Human](https://

EasyMocap

Install / Use

README