DualStyleGAN - Official PyTorch Implementation

<img src="./doc_images/overview.jpg" width="96%" height="96%">

This repository provides the official PyTorch implementation for the following paper:

Pastiche Master: Exemplar-Based High-Resolution Portrait Style Transfer<br> Shuai Yang, Liming Jiang, Ziwei Liu and Chen Change Loy<br> In CVPR 2022.<br> Project Page | Paper | Supplementary Video

Abstract: Recent studies on StyleGAN show high performance on artistic portrait generation by transfer learning with limited data. In this paper, we explore more challenging exemplar-based high-resolution portrait style transfer by introducing a novel <b>DualStyleGAN</b> with flexible control of dual styles of the original face domain and the extended artistic portrait domain. Different from StyleGAN, DualStyleGAN provides a natural way of style transfer by characterizing the content and style of a portrait with an <b>intrinsic style path</b> and a new <b>extrinsic style path</b>, respectively. The delicately designed extrinsic style path enables our model to modulate both the color and complex structural styles hierarchically to precisely pastiche the style example. Furthermore, a novel progressive fine-tuning scheme is introduced to smoothly transform the generative space of the model to the target domain, even with the above modifications on the network architecture. Experiments demonstrate the superiority of DualStyleGAN over state-of-the-art methods in high-quality portrait style transfer and flexible style control.

Features:<br> High-Resolution (1024) | Training Data-Efficient (~200 Images) | Exemplar-Based Color and Structure Transfer

Updates

• [02/2023] Add --wplus in style_transfer.py to use original w+ pSp encoder rather than z+.
• [09/2022] Pre-trained models in three new styles (feat. StableDiffusion) are released.
• [07/2022] Source code license is updated.
• [03/2022] Paper and supplementary video are released.
• [03/2022] Web demo is created.
• [03/2022] Code is released.
• [03/2022] This website is created.

Web Demo

Integrated into Huggingface Spaces 🤗 using Gradio. Try out the Web Demo: or

Installation

Clone this repo:

git clone https://github.com/williamyang1991/DualStyleGAN.git
cd DualStyleGAN

Dependencies:

All dependencies for defining the environment are provided in environment/dualstylegan_env.yaml. We recommend running this repository using Anaconda:

conda env create -f ./environment/dualstylegan_env.yaml

We use CUDA 10.1 so it will install PyTorch 1.7.1 (corresponding to Line 22, Line 25, Line 26 of dualstylegan_env.yaml). Please install PyTorch that matches your own CUDA version following https://pytorch.org/.

☞ Install on Windows: here and here

(1) Dataset Preparation

Cartoon, Caricature and Anime datasets can be downloaded from their official pages. We also provide the script to build new datasets.

| Dataset | Description | | :--- | :--- | | Cartoon | 317 cartoon face images from Toonify. | | Caricature | 199 images from WebCaricature. Please refer to dataset preparation for more details. | | Anime | 174 images from Danbooru Portraits. Please refer to dataset preparation for more details. | | Fantasy | 137 fantasy face images generated by StableDiffusion. | | Illustration | 156 illustration face images generated by StableDiffusion. | | Impasto | 120 impasto face images generated by StableDiffusion. | | Other styles | Please refer to dataset preparation for the way of building new datasets. |

<br/>

(2) Inference for Style Transfer and Artistic Portrait Generation

Inference Notebook

<a href="http://colab.research.google.com/github/williamyang1991/DualStyleGAN/blob/master/notebooks/inference_playground.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" height=22.5></a>
To help users get started, we provide a Jupyter notebook found in ./notebooks/inference_playground.ipynb that allows one to visualize the performance of DualStyleGAN. The notebook will download the necessary pretrained models and run inference on the images found in ./data/.

If no GPU is available, you may refer to Inference on CPU, and set device = 'cpu' in the notebook.

Pretrained Models

Pretrained models can be downloaded from Google Drive or Baidu Cloud (access code: cvpr):

| Model | Description | | :--- | :--- | | encoder | Pixel2style2pixel encoder that embeds FFHQ images into StyleGAN2 Z+ latent code | | encoder_wplus | Original Pixel2style2pixel encoder that embeds FFHQ images into StyleGAN2 W+ latent code | | cartoon | DualStyleGAN and sampling models trained on Cartoon dataset, 317 (refined) extrinsic style codes | | caricature | DualStyleGAN and sampling models trained on Caricature dataset, 199 (refined) extrinsic style codes | | anime | DualStyleGAN and sampling models trained on Anime dataset, 174 (refined) extrinsic style codes | | arcane | DualStyleGAN and sampling models trained on Arcane dataset, 100 extrinsic style codes | | comic | DualStyleGAN and sampling models trained on Comic dataset, 101 extrinsic style codes | | pixar | DualStyleGAN and sampling models trained on Pixar dataset, 122 extrinsic style codes | | slamdunk | DualStyleGAN and sampling models trained on Slamdunk dataset, 120 extrinsic style codes | | fantasy | DualStyleGAN models trained on Fantasy dataset, 137 extrinsic style codes | | illustration | DualStyleGAN models trained on Illustration dataset, 156 extrinsic style codes | | impasto | DualStyleGAN models trained on Impasto dataset, 120 extrinsic style codes |

The saved checkpoints are under the following folder structure:

checkpoint
|--encoder.pt                     % Pixel2style2pixel model
|--encoder_wplus.pt               % Pixel2style2pixel model (optional)
|--cartoon
    |--generator.pt               % DualStyleGAN model
    |--sampler.pt                 % The extrinsic style code sampling model
    |--exstyle_code.npy           % extrinsic style codes of Cartoon dataset
    |--refined_exstyle_code.npy   % refined extrinsic style codes of Cartoon dataset
|--caricature
    % the same files as in Cartoon
...

Exemplar-Based Style Transfer

Transfer the style of a default Cartoon image onto a default face:

python style_transfer.py 

The result cartoon_transfer_53_081680.jpg is saved in the folder .\output\, where 53 is the id of the style image in the Cartoon dataset, 081680 is the name of the content face image. An corresponding overview image cartoon_transfer_53_081680_overview.jpg is additionally saved to illustrate the input content image, the encoded content image, the style image (* the style image will