🖼️ ImagenHub

ImagenHub: Standardizing the evaluation of conditional image generation models <br> ICLR 2024

ImagenHub is a one-stop library to standardize the inference and evaluation of all the conditional image generation models.

We define 7 prominent tasks and curate 7 high-quality evaluation datasets for each task.
We built a unified inference pipeline to ensure fair comparison. We currently support over 50 models.
We designed two human evaluation scores, i.e. Semantic Consistency and Perceptual Quality, along with comprehensive guidelines to evaluate generated images.
We provide code for visualization, autometrics and Amazon mechanical turk templates.

<div align="center"> <a href = "https://tiger-ai-lab.github.io/ImagenHub/">[🌐 Project Page]</a> <a href = "https://imagenhub.readthedocs.io/en/latest/index.html">[📘 Documentation]</a> <a href = "https://arxiv.org/abs/2310.01596">[📄 Arxiv]</a> <a href = "https://huggingface.co/ImagenHub">[💾 Datasets]</a> <a href = "https://chromaica.github.io/#imagen-museum">[🏛️ ImagenMuseum]</a> <a href = "https://huggingface.co/spaces/TIGER-Lab/GenAI-Arena">[⚔️ GenAI-Arena]</a> </div>

📰 News

2024 Jun 10: GenAI-Arena Paper is out. It is featured on Huggingface Daily Papers.
2024 Jun 07: ImagenHub is finally on PyPI! Check: https://pypi.org/project/imagen-hub/
2024 Apr 07: We released Human evaluation ratings According to our latest Arxiv paper.
2024 Feb 14: Checkout ⚔️ GenAI-Arena ⚔️ : Benchmarking Visual Generative Models in the Wild!
2024 Jan 15: Paper accepted to ICLR 2024! See you in Vienna!
2024 Jan 7: We updated Human Evaluation Guideline, ImagenMuseum Submission! Now we welcome researchers to submit their method on ImagenMuseum with minimal effort.
2023 Oct 23: Version 0.1.0 released! ImagenHub’s documentation now available!
2023 Oct 19: Code Released. Docs under construction.
2023 Oct 13: We released Imagen Museum, a visualization page for all models from ImagenHub!
2023 Oct 4: Our paper is featured on Huggingface Daily Papers!
2023 Oct 2: Paper available on Arxiv. Code coming Soon!

🛠️ Installation 🔝

Install from PyPI:

pip install imagen-hub

Or build from source:

git clone https://github.com/TIGER-AI-Lab/ImagenHub.git
cd ImagenHub
conda env create -f env_cfg/imagen_environment.yml
conda activate imagen
pip install -e .

For models like Dall-E, DreamEdit, and BLIPDiffusion, please see Extra Setup

For some models (Stable Diffusion, SDXL, CosXL, etc.), you need to login through huggingface-cli.

huggingface-cli login

👨‍🏫 Get Started 🔝

Benchmarking

To reproduce our experiment reported in the paper:

Example for text-guided image generation:

python3 benchmarking.py -cfg benchmark_cfg/ih_t2i.yml

Note that the expected output structure would be:

result_root_folder
└── experiment_basename_folder
    ├── input (If applicable)
    │   └── image_1.jpg ...
    ├── model1
    │   └── image_1.jpg ...
    ├── model2
    │   └── image_1.jpg ...
    ├── ...

Then after running the experiment, you can run

python3 visualize.py --cfg benchmark_cfg/ih_t2i.yml

to produce a index.html file for visualization.

The file would look like something like this. We hosted our experiment results on Imagen Museum. <img src="https://i.imgur.com/0uOMhtT.png" width="512" />

Infering one model

import imagen_hub

model = imagen_hub.load("SDXL")
image = model.infer_one_image(prompt="people reading pictures in a museum, watercolor", seed=1)
image

Running Metrics

from imagen_hub.metrics import MetricLPIPS
from imagen_hub.utils import load_image, save_pil_image, get_concat_pil_images

def evaluate_one(model, real_image, generated_image):
  score = model.evaluate(real_image, generated_image)
  print("====> Score : ", score)

image_I = load_image("https://chromaica.github.io/Museum/ImagenHub_Text-Guided_IE/input/sample_102724_1.jpg")
image_O = load_image("https://chromaica.github.io/Museum/ImagenHub_Text-Guided_IE/DiffEdit/sample_102724_1.jpg")
show_image = get_concat_pil_images([image_I, image_O], 'h')

model = MetricLPIPS()
evaluate_one(model, image_I, image_O) # ====> Score :  0.11225218325853348

show_image

📘 Documentation 🔝

The tutorials and API documentation are hosted on imagenhub.readthedocs.io.

🧠 Philosophy 🔝

By streamlining research and collaboration, ImageHub plays a pivotal role in propelling the field of Image Generation and Editing.

Purity of Evaluation: We ensure a fair and consistent evaluation for all models, eliminating biases.
Research Roadmap: By defining tasks and curating datasets, we provide clear direction for researchers.
Open Collaboration: Our platform fosters the exchange and cooperation of related technologies, bringing together minds and innovations.

Implemented Models

We included more than 30 Models in image synthesis. See the full list here:

Supported Models: https://github.com/TIGER-AI-Lab/ImagenHub/issues/1
Supported Metrics: https://github.com/TIGER-AI-Lab/ImagenHub/issues/6

| Method |:-------------------- | Stable Diffusion | Stable Diffusion XL | DeepFloyd-IF | OpenJourney | Dall-E | Kandinsky | MagicBrush | InstructPix2Pix | DiffEdit | Imagic | CycleDiffusion | SDEdit | Prompt-to-Prompt | Text2Live | Pix2PixZero | GLIDE | Blended Diffusion | Stable Diffusion Inpainting | | Stable Diffusion XL Inpainting | | TextualInversion | BLIP-Diffusion | DreamBooth(+ LoRA) | Photoswap | DreamEdit | Custom Diffusion | ControlNet | UniControl | Venue | Type | -------:|:-------------:|:-------------------------: | | - | Text-To-Image Generation | | arXiv'23 | Text-To-Image Generation | | - | Text-To-Image Generation | | - | Text-To-Image Generation | | - | Text-To-Image Generation | | - | Text-To-Image Generation | | arXiv'23 | Text-guided Image Editing | | CVPR'23 | Text-guided Image Editing | | ICLR'23 | Text-guided Image Editing | | CVPR'23 | Text-guided Image Editing | | ICCV'23 | Text-guided Image Editing | | ICLR'22 | Text-guided Image Editing | | ICLR'23 | Text-guided Image Editing | | ECCV'22 | Text-guided Image Editing | | SIGGRAPH'23 | Text-guided Image Editing | | ICML'22 | Mask-guided Image Editing | | CVPR'22 | Mask-guided Image Editing | - | Mask-guided Image Editing | - | Mask-guided Image Editing | | ICLR'23 | Subject-driven Image Generation| | arXiv'23 | Subject-Driven Image Generation| | CVPR'23 | Subject-Driven Image Generation| | arXiv'23 | Subject-Driven Image Editing | | arXiv'23 | Subject-Driven Image Editing | | CVPR'23 | Multi-Subject-Driven Generation| | arXiv'23 | Control-guided Image Generation| | arXiv'23 | Control-guided Image Generation|

ImagenHub

Install / Use

README