ObjectCompose: Evaluating Resilience of Vision-Based Models on Object-to-Background Compositional Changes [ACCV 2024 Oral]

Best Student Paper - Runner up

Hashmat Shadab Malik*, Muhammad Huzaifa*, Muzammal Naseer, Salman Khan, and Fahad Shahbaz Khan

<hr />

:rocket: News

• (September 25, 2024)
  • Paper selected for Oral presentation at ACCV 2024!🎊
• (September 20, 2024)
  • Paper accepted at ACCV 2024!🎊
• (March 15, 2024)
  • Datasets generated by ObjectCompose on ImageNet and COCO are released.
  • Pre-trained models and evaluation code is released.
  • Code for ObjectCompose is released.

<hr />

Abstract: Given the large-scale multi-modal training of recent vision-based models and their generalization capabilities, understanding the extent of their robustness is critical for their real-world deployment. In this work, we evaluate the resilience of current vision-based models against diverse object-to-background context variations. The majority of robustness evaluation methods have introduced synthetic datasets to induce changes to object characteristics (viewpoints, scale, color) or utilized image transformation techniques (adversarial changes, common corruptions) on real images to simulate shifts in distributions. Recent works have explored leveraging large language models and diffusion models to generate changes in the background. However, these methods either lack in offering control over the changes to be made or distort the object semantics, making them unsuitable for the task. Our method, on the other hand, can induce diverse object-to-background changes while preserving the original semantics and appearance of the object. To achieve this goal, we harness the generative capabilities of text-to-image, image-to-text, and image-to-segment models to automatically generate a broad spectrum of object-to-background changes. We induce both natural and adversarial background changes by either modifying the textual prompts or optimizing the latents and textual embedding of text-to-image models. This allows us to quantify the role of background context in understanding the robustness and generalization of deep neural networks. We produce various versions of standard vision datasets (ImageNet, COCO), incorporating either diverse and realistic backgrounds into the images or introducing color, texture, and adversarial changes in the background. We conduct extensive experiment to analyze the robustness of vision-based models against object-to-background context variations across diverse tasks.

Contents

1. Highlights
2. Installation
3. Dataset Preparation
4. Background Generation
5. Datasets
6. Evaluation
7. Results

<hr> <hr>

Highlights

^(top)

ObjectCompose consists of an inpainting-based diffusion model to generate the counterfactual background of an image. The object mask is obtained from a segmentation model (SAM) by providing the class label as an input prompt. The segmentation mask, along with the original image caption (generated via BLIP-2) is then processed through the diffusion model. For generating adversarial examples, both the latent and conditional embedding are optimized during the denoising process.

Installation

^(top)

conda create -n objcomp python==3.11.3
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia
pip install -r requirements.txt
pip install git+https://github.com/openai/CLIP.git
pip install git+https://github.com/huggingface/transformers.git

diffusers==0.18.2 is used in this codebase.

Dataset Preparation

^(top)

Please find data preparation instructions in DataPrep.md.

Background Generation

^(top)

On ImageNet dataset, run the following command to generate the background changes:

python obj_compose.py --dataset imagenet --data_path </path/to/image/folder> --save_dir </path/to/save/generated/dataset> --diffusion_steps <> --guidance <> --background_change <>  

This would generate the new dataset in the ImageNet format and save it in the save_dir. background_change can be set to caption, class_name, prompt

For Adversarial Background Generation, run the following command

python obj_compose_adv.py --dataset <imagenet or coco> --data_path </path/to/image/folder> --save_dir </path/to/save/generated/dataset> --diffusion_steps <> --start_step <> --background_change <>  --attack_type <>

This would generate the new dataset in the ImageNet or COCO format and save it in the save_dir. attack_type can be set to text, latent, or ensemble

On COCO dataset, run the following command to generate the background changes:

python obj_compose.py --dataset coco --data_path <> --save_dir <> --diffusion_steps <> --guidance <> --prompt <>  

This would generate the new dataset in the COCO format and save it in the save_dir. background_change can be set to caption, prompt

• --images_per_class: This argument is used for the ImageNet and COCO_Classification datasets to specify the number of images per class.
• --expand_mask_pixels: This argument can be used for all the datasets to expand the size of the mask covering the object of interest.
• --prompt: This argument determines whether to use a manual prompt.

If you want to use a prompt and the class name together, pass the --prompt argument in the following format: --prompt="A photo of a XXX in a red background" .XXX will be replaced by the class name.

Datasets

^(top)

Background Generation by ObjectCompose

<table> <tr> <th>ImageNet-B</th> <th>ImageNet-B_1k</th> <th>COCO-DC (Classification)</th> <th>COCO-DC (Detection)</th> </tr> <tr> <td>

| Category | Link | |:-------------|:----:| | Original | Link | | Class Name | Link | | BLiP Caption | Link | | Color | Link | | Texture | Link | | Adversarial | Link |

</td> <td>

| Category | Link | |:-------------|:----:| | Original | Link | | Class Name | Link | | BLiP Caption | Link | | Color | Link | | Texture | Link | | Adversarial | Link |

</td> <td>

| Category | Link | |:-------------|:----:| | Original | Link | | BLiP Caption | Link | | Color | Link | | Texture | Link | | Adversarial | Link |

</td> <td>

| Category | Link | |:-------------|:----:| | Original | Link | | BLiP Caption | Link | | Color | Link | | Texture | Link | | Adversarial | Link |

</td> </tr> </table>

Background Generation by Other Methods on ImageNet-B_1k dataset

<table> <tr valign="top"> <td>

1. ImageNet-E (CVPR-2023)

| Category | Link | |:----------------------------|:---------------:| | Original | Link | | ImageNet-E ($λ=-20$) | [Li