Adaptive Affine Transformation: A Simple and Effective Operation for Spatial Misaligned Image Generation. (accepted in MM2022)

Paper Demo video Supplementary materials

Face Reenactment & Talking Face

The 3DMM model can not be released, we will replace it with FLAME in the future.

Face visually dubbing

The source code is in here.

Person image generation

Download resources (pretrained model etc.)

Download resources in Google drive, including:

• person_epoch_30.pth: Pretrained model on deep fashion dataset stopped in 30 epoch.
• person_epoch_40.pth: Pretrained model on deep fashion dataset stopped in 40 epoch (has better performance than "person_epoch_30.pth").
• test_image_person_deepFashion_30epoch.zip: Inference images of "person_epoch_30.pth" on deep fashion test data for convenient comparisons.
• test_image_person_deepFashion_40epoch.zip: Inference images of "person_epoch_40.pth" on deep fashion test data for convenient comparisons.
• example_person_source_img.jpg: Source example image for person image generation.
• example_person_souce_kp.txt: Source example key points for person image generation.
• example_person_target_kp.txt: Target example key points for person image generation.
• example_person_inference_img.jpg: Inference example image for person image generation.
• fasion_train_data.json: Training json file of deep fashion dataset.\

Train on deep fashion dataset

1. Download deep fashion dataset from here. We use the dataset as same as in SelectionGAN.
2. Unzip the dataset.
3. run

python train_person_image.py --train_data=./assert/fasion_train_data.json --train_img_dir=./deepFashion/fashion_data/train

Inference

To inference one person image from one source person image, source key points and target key points, run

python inference_person_image.py --inference_model_path=./assert/person_epoch_30.pth --source_img_path=./assert/example_person_source_img.jpg --source_kp_path=./assert/example_person_souce_kp.txt --target_kp_path=./assert/example_person_target_kp.txt --res_person_path=./assert/example_person_inference_img.jpg

Compute metrics

To compute the metrics of SSIM and LIPIS on deep fashion test data, run

python compute_metrics.py --inference_img_dir --real_img_dir=./deepFashion/fashion_data/test --task_type=person

Citation

If you use AdaAT operator in your work, please cite

@inproceedings{zhang2022adaptive,
  title={Adaptive Affine Transformation: A Simple and Effective Operation for Spatial Misaligned Image Generation},
  author={Zhang, Zhimeng and Ding, Yu},
  booktitle={Proceedings of the 30th ACM International Conference on Multimedia},
  pages={1167--1176},
  year={2022}
}

Acknowledgement

The basic modules are borrowed from first-order-model, thanks for their contributions.