Mimicry

---

About | Documentation | Tutorial | Gallery | Paper

Mimicry is a lightweight PyTorch library aimed towards the reproducibility of GAN research.

Comparing GANs is often difficult - mild differences in implementations and evaluation methodologies can result in huge performance differences. Mimicry aims to resolve this by providing: (a) Standardized implementations of popular GANs that closely reproduce reported scores; (b) Baseline scores of GANs trained and evaluated under the same conditions; (c) A framework for researchers to focus on implementation of GANs without rewriting most of GAN training boilerplate code, with support for multiple GAN evaluation metrics.

We provide a model zoo and set of baselines to benchmark different GANs of the same model size trained under the same conditions, using multiple metrics. To ensure reproducibility, we verify scores of our implemented models against reported scores in literature.

---

Installation

The library can be installed with:

pip install git+https://github.com/kwotsin/mimicry.git

See also setup information for more.

Example Usage

Training a popular GAN like SNGAN that reproduces reported scores can be done as simply as:

import torch
import torch.optim as optim
import torch_mimicry as mmc
from torch_mimicry.nets import sngan

# Data handling objects
device = torch.device('cuda:0' if torch.cuda.is_available() else "cpu")
dataset = mmc.datasets.load_dataset(root='./datasets', name='cifar10')
dataloader = torch.utils.data.DataLoader(
    dataset, batch_size=64, shuffle=True, num_workers=4)

# Define models and optimizers
netG = sngan.SNGANGenerator32().to(device)
netD = sngan.SNGANDiscriminator32().to(device)
optD = optim.Adam(netD.parameters(), 2e-4, betas=(0.0, 0.9))
optG = optim.Adam(netG.parameters(), 2e-4, betas=(0.0, 0.9))

# Start training
trainer = mmc.training.Trainer(
    netD=netD,
    netG=netG,
    optD=optD,
    optG=optG,
    n_dis=5,
    num_steps=100000,
    lr_decay='linear',
    dataloader=dataloader,
    log_dir='./log/example',
    device=device)
trainer.train()

# Evaluate fid
mmc.metrics.evaluate(
    metric='fid',
    log_dir='./log/example',
    netG=netG,
    dataset='cifar10',
    num_real_samples=50000,
    num_fake_samples=50000,
    evaluate_step=100000,
    device=device)

Example outputs:

>>> INFO: [Epoch 1/127][Global Step: 10/100000]
| D(G(z)): 0.5941
| D(x): 0.9303
| errD: 1.4052
| errG: -0.6671
| lr_D: 0.0002
| lr_G: 0.0002
| (0.4550 sec/idx)
^CINFO: Saving checkpoints from keyboard interrupt...
INFO: Training Ended

Tensorboard visualizations:

tensorboard --logdir=./log/example

See further details in example script, as well as a detailed tutorial on implementing a custom GAN from scratch.

Further Guides

• Evaluating a pre-trained generator model
• Evaluation using custom datasets
• Implementing, training and evaluating a custom GAN

<div id="baselines"></div>

Baselines | Model Zoo

For a fair comparison, we train all models under the same training conditions for each dataset, each implemented using ResNet backbones of the same architectural capacity. We train our models with the Adam optimizer using the popular hyperparameters (β₁, β₂) = (0.0, 0.9). n_dis represents the number of discriminator update steps per generator update step, and n_iter is simply the number of training iterations.

Models

| Abbrev. | Name | Type* | |:-----------:|:---------------------------------------------:|:-------------:| | DCGAN | Deep Convolutional GAN | Unconditional | | WGAN-GP | Wasserstein GAN with Gradient Penalty | Unconditional | | SNGAN | Spectral Normalization GAN | Unconditional | | cGAN-PD | Conditional GAN with Projection Discriminator | Conditional | | SSGAN | Self-supervised GAN | Unconditional | | InfoMax-GAN | Infomax-GAN | Unconditional |

*Conditional GAN scores are only reported for labelled datasets.

Metrics

| Metric | Method | |:--------------------------------:|:---------------------------------------:| | Inception Score (IS)* | 50K samples at 10 splits| | Fréchet Inception Distance (FID) | 50K real/generated samples | | Kernel Inception Distance (KID) | 50K real/generated samples, averaged over 10 splits.|

*Inception Score can be a poor indicator of GAN performance, as it does not measure diversity and is not domain agnostic. This is why certain datasets with only a single class (e.g. CelebA and LSUN-Bedroom) will perform poorly when using this metric.

Datasets

| Dataset | Split | Resolution | |:------------:|:---------:|:----------:| | CIFAR-10 | Train | 32 x 32 | | CIFAR-100 | Train | 32 x 32 | | ImageNet | Train | 32 x 32 | | STL-10 | Unlabeled | 48 x 48 | | CelebA | All | 64 x 64 | | CelebA | All | 128 x 128 | | LSUN-Bedroom | Train | 128 x 128 | | ImageNet | Train | 128 x 128 |

---

CelebA

Paper | Dataset

Training Parameters

| Resolution | Batch Size | Learning Rate | β₁ | β₂ | Decay Policy | n_dis | n_iter | |:----------:|:----------:|:-------------:|:-------------:|:-------------:|:------------:|:---------------:|------------------| | 128 x 128 | 64 | 2e-4 | 0.0 | 0.9 | None | 2 | 100K | | 64 x 64 | 64 | 2e-4 | 0.0 | 0.9 | Linear | 5 | 100K |

Results

| Resolution | Model | IS | FID | KID | Checkpoint | Code | |:---:|:---:|:---:|:---:|:---:|:---:|:---:| | 128 x 128 | SNGAN | 2.72 ± 0.01 | 12.93 ± 0.04 | 0.0076 ± 0.0001 | netG.pth | sngan_128.py | | 128 x 128 | SSGAN | 2.63 ± 0.01 | 15.18 ± 0.10 | 0.0101 ± 0.0001 | netG.pth | ssgan_128.py | | 128 x 128 | InfoMax-GAN | 2.84 ± 0.01 | 9.50 ± 0.04 | 0.0063 ± 0.0001 | netG.pth | infomax_gan_128.py | | 64 x 64 | SNGAN | 2.68 ± 0.01 | 5.71 ± 0.02 | 0.0033 ± 0.0001 | netG.pth | sngan_64.py | | 64 x 64 | SSGAN | 2.67 ± 0.01 | 6.03 ± 0.04 | 0.0036 ± 0.0001 | netG.pth | ssgan_64.py | | 64 x 64 | InfoMax-GAN |2.68 ± 0.01 | 5.71 ± 0.06 | 0.0033 ± 0.0001 | netG.pth | infomax_gan_64.py |

LSUN-Bedroom

Paper | Dataset

Training Parameters

| Resolution | Batch Size | Learning Rate | β₁ | β₂ | Decay Policy | n_dis | n_iter | |:----------:|:----------:|:-------------:|:-------------:|:-------------:|:------------:|:---------------:|------------------| | 128 x 128 | 64 | 2e-4 | 0.0 | 0.9 | Linear | 2 | 100K |

Results

| Resolution | Model | IS | FID | KID | Checkpoint | Code | |:---:|:---:|:---:|:---:|:---:|:---:|:---:| | 128 x 128 | SNGAN | 2.30 ± 0.01 | 25.87 ± 0.03 | 0.0141 ± 0.0001 | netG.pth | sngan_128.py | | 128 x 128 | SSGAN | 2.12 ± 0.01 | 12.02 ± 0.07 | 0.0077 ± 0.0001 | netG.pth | ssgan_128.py | | 128 x 128 | InfoMax-GAN |2.22 ± 0.01 | 12.13 ± 0.16 | 0.0080 ± 0.0001 | netG.pth | infomax_gan_128.py |

STL-10

Paper | Dataset

Training Parameters

| Resolution | Batch Size | Learning Rate | β₁ | β₂ | Decay Policy | n_dis | n_iter | |:----------:|:----------:|:-------------:|:-------------:|:-------------:|:------------:|:---------------:|------------------| | 48 x 48 | 64