Talking Head Anime from a Single Image 3

A Python package to more easily install pkhungurn/talking-head-anime-3-demo and use Python APIs. The package ships with the officially distributed seperated_float model (only) by default.

Installation

Install this via pipx or pip (or your favourite package manager):

pipx install tha3
pipx inject tha3 torch --index-url https://download.pytorch.org/whl/cu118 --pip-args="--upgrade"

Usage

tha3 # manual poser
tha3i # iFacialMocap Puppeteer

Contributors ✨

Thanks goes to these wonderful people (emoji key):

<table> <tbody> <tr> <td align="center" valign="top" width="14.28%"><a href="https://github.com/34j"><img src="https://avatars.githubusercontent.com/u/55338215?v=4?s=80" width="80px;" alt="34j"/><br /><sub><b>34j</b></sub></a><br /><a href="https://github.com/34j/tha3/commits?author=34j" title="Code">💻</a> <a href="#ideas-34j" title="Ideas, Planning, & Feedback">🤔</a> <a href="https://github.com/34j/tha3/commits?author=34j" title="Documentation">📖</a></td> </tr> </tbody> </table>

This project follows the all-contributors specification. Contributions of any kind welcome!

Demo Code for "Talking Head(?) Anime from A Single Image 3: Now the Body Too"

This repository contains demo programs for the Talking Head(?) Anime from a Single Image 3: Now the Body Too project. As the name implies, the project allows you to animate anime characters, and you only need a single image of that character to do so. There are two demo programs:

The manual_poser lets you manipulate a character's facial expression, head rotation, body rotation, and chest expansion due to breathing through a graphical user interface.
ifacialmocap_puppeteer lets you transfer your facial motion to an anime character.

Try the Manual Poser on Google Colab

If you do not have the required hardware (discussed below) or do not want to download the code and set up an environment to run it, click to try running the manual poser on Google Colab.

Hardware Requirements

Both programs require a recent and powerful Nvidia GPU to run. I could personally ran them at good speed with the Nvidia Titan RTX. However, I think recent high-end gaming GPUs such as the RTX 2080, the RTX 3080, or better would do just as well.

The ifacialmocap_puppeteer requires an iOS device that is capable of computing blend shape parameters from a video feed. This means that the device must be able to run iOS 11.0 or higher and must have a TrueDepth front-facing camera. (See this page for more info.) In other words, if you have the iPhone X or something better, you should be all set. Personally, I have used an iPhone 12 mini.

Software Requirements

GPU Related Software

Please update your GPU's device driver and install the CUDA Toolkit that is compatible with your GPU and is newer than the version you will be installing in the next subsection.

Python Environment

Both manual_poser and ifacialmocap_puppeteer are available as desktop applications. To run them, you need to set up an environment for running programs written in the Python language. The environment needs to have the following software packages:

Python >= 3.8
PyTorch >= 1.11.0 with CUDA support
SciPY >= 1.7.3
wxPython >= 4.1.1
Matplotlib >= 3.5.1

One way to do so is to install Anaconda and run the following commands in your shell:

> conda create -n talking-head-anime-3-demo python=3.8
> conda activate talking-head-anime-3-demo
> conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
> conda install scipy
> pip install wxpython
> conda install matplotlib

Caveat 1: Do not use Python 3.10 on Windows

As of June 2006, you cannot use wxPython with Python 3.10 on Windows. As a result, do not use Python 3.10 until this bug is fixed. This means you should not set python=3.10 in the first conda command in the listing above.

Caveat 2: Adjust versions of Python and CUDA Toolkit as needed

The environment created by the commands above gives you Python version 3.8 and an installation of PyTorch that was compiled with CUDA Toolkit version 11.3. This particular setup might not work in the future because you may find that this particular PyTorch package does not work with your new computer. The solution is to:

Change the Python version in the first command to a recent one that works for your OS. (That is, do not use 3.10 if you are using Windows.)
Change the version of CUDA toolkit in the third command to one that the PyTorch's website says is available. In particular, scroll to the "Install PyTorch" section and use the chooser there to pick the right command for your computer. Use that command to install PyTorch instead of the third command above.

The command to install PyTorch

Jupyter Environment

The manual_poser is also available as a Jupyter Nootbook. To run it on your local machines, you also need to install:

Jupyter Notebook >= 7.3.4
IPywidgets >= 7.7.0

In some case, you will also need to enable the widgetsnbextension as well. So, run

> jupyter nbextension enable --py widgetsnbextension

After installing the above two packages. Using Anaconda, I managed to do the above with the following commands:

> conda install -c conda-forge notebook
> conda install -c conda-forge ipywidgets
> jupyter nbextension enable --py widgetsnbextension

Automatic Environment Construction with Anaconda

You can also use Anaconda to download and install all Python packages in one command. Open your shell, change the directory to where you clone the repository, and run:

> conda env create -f environment.yml

This will create an environment called talking-head-anime-3-demo containing all the required Python packages.

iFacialMocap

If you want to use ifacialmocap_puppeteer, you will also need to an iOS software called iFacialMocap (a 980 yen purchase in the App Store). You do not need to download the paired application this time. Your iOS and your computer must use the same network. For example, you may connect them to the same wireless router.

Tha3

Install / Use

README