ViCo

ViCo: Detail-Preserving Visual Condition for Personalized Text-to-Image Generation

⏳ To Do

• [x] Release inference code
• [x] Release pretrained models
• [x] Release training code
• [ ] Quantitative evaluation code
• [ ] Hugging Face demo

⚙️ Set-up

Create a conda environment vico using

conda env create -f environment.yaml
conda activate vico

⏬ Download

Download the pretrained stable diffusion v1-4 under models/ldm/stable-diffusion-v1.

We provide the pretrained checkpoints at 300, 350, and 400 steps of 8 objects. You can download the sample images and their corresponding pretrained checkpoints. You can also download the data of any object:

| Object | Sample images | Checkpoints | | :---- | :----: | :----: | | barn | image | ckpt | | batman | image | ckpt | | clock | image | ckpt | | dog7 | image | ckpt | | monster toy | image | ckpt | | pink sunglasses | image | ckpt | | teddybear | image | ckpt | | wooden pot | image | ckpt |

Datasets are originally collected and provided by Textual Inversion, DreamBooth, and Custom Diffsuion. You can find all datasets used for quantitaive comparison in our paper.

🚀 Inference

Before running the inference command, please set:

• REF_IMAGE_PATH: Path of the reference image. It can be any image in the samples like batman/1.jpg.
• CHECKPOINT_PATH: Path of the checkpoint weight. Its subfolder should be similar to checkpoints/*-399.pt.
• OUTPUT_PATH: Path of the generated images. For example, it can be like outputs/batman.

python scripts/vico_txt2img.py \
--ddim_eta 0.0  --n_samples 4  --n_iter 2  --scale 7.5  --ddim_steps 50  \
--ckpt_path models/ldm/stable-diffusion-v1/sd-v1-4.ckpt  \
--image_path REF_IMAGE_PATH \
--ft_path CHECKPOINT_PATH \
--load_step 399 \
--prompt "a photo of * on the beach" \
--outdir OUTPUT_PATH

You can specify load_step (300,350,400) and personalize prompt (a prefix "a photo of" usually makes better results).

💻 Training

Before running the training command, please set:

• RUN_NAME: Your run name. Will be the name of the folder of logs.
• GPUS_USED: GPUs you are using, e.g., "0,1,2,3". (4 RTX 3090 GPUs in my case)
• TRAIN_DATA_ROOT: Path of your training images.
• INIT_WORD: Initialize the word to represent your unique object, e.g., "dog" and "toy".

python main.py \
--base configs/stable-diffusion/v1-finetune.yaml -t  \
--actual_resume models/ldm/stable-diffusion-v1/sd-v1-4.ckpt  \
-n RUN_NAME \
--gpus  GPUS_USED \
--data_root TRAIN_DATA_ROOT \
--init_word INIT_WORD

📖 Citation

If you use this code in your research, please consider citing our paper:

@inproceedings{Hao2023ViCo,
  title={ViCo: Detail-Preserving Visual Condition for Personalized Text-to-Image Generation},
  author={Shaozhe Hao and Kai Han and Shihao Zhao and Kwan-Yee K. Wong},
  year={2023}
}

💐 Acknowledgements

This code repository is based on the great work of Textual Inversion. Thanks!