ImagenetX : Understanding model mistakes with factors of variation annotations

Code to load annotations, evaluate models and reproduce paper plots. See [paper], [website], [colab]

Installation

If you just want to load the annotations :

pip install imagenet-x

Other installations:

• Install from repo clone : pip install -e .
• Reproduce plots from the paper and use dataset loader: pip install 'imagenet-x[all]'

Usage

To load the annotations

from imagenet_x import load_annotations

annotations = load_annotations()

This will output the following table

| file_name | class | multiple_objects | background | color | brighter | darker | style | larger | smaller | object_blocking | person_blocking | partial_view | pattern | pose | shape | subcategory | texture | justification | one_word | metaclass | |:-----------------------------|--------:|-------------------:|-------------:|--------:|-----------:|---------:|--------:|---------:|----------:|------------------:|------------------:|---------------:|----------:|-------:|--------:|--------------:|----------:|:------------------------------------------------|:------------------|:------------| | ILSVRC2012_val_00004487.JPEG | 762 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | close up of a pan fried sea bass. | sea bass close up | structure | | ILSVRC2012_val_00003963.JPEG | 292 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | sepia image of tiger | digitally altered | other | | ILSVRC2012_val_00041992.JPEG | 718 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | the bridge is brown | rare view | device | | ILSVRC2012_val_00028056.JPEG | 635 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | the magnetic compass is on the bronze container | wood shape | device |

See this notebook for some sample images and annotations

One can also directly download the raw annotation files stored in the annotations folder. There are 4 json line files imagenet_x_[train|val]_[multi|top]_factor.jsonl that have entries such as the following:

{"file_name":"ILSVRC2012_val_00004487.JPEG","class":762,"multiple_objects":0,"background":1,"color":0,"brighter":0,"darker":0,"style":0,"larger":1,"smaller":0,"object_blocking":0,"person_blocking":0,"partial_view":0,"pattern":0,"pose":1,"shape":0,"subcategory":0,"texture":0,"justification":"close up of a pan fried sea bass. ","one_word":"sea bass close up"}

To generate plots for a new model

Generate the predictions of your model on the Imagenet Validation set in a csv file with 3 columns

| file_name | predicted_class | predicted_probability | |:-----------------------------|------------------:|------------------------:| | ILSVRC2012_val_00000293.JPEG | 0 | 0.634764 | | ILSVRC2012_val_00002138.JPEG | 391 | 0.360206 | | ILSVRC2012_val_00003014.JPEG | 0 | 0.951837 | | ILSVRC2012_val_00006697.JPEG | 0 | 0.999731 | | ILSVRC2012_val_00007197.JPEG | 0 | 0.998473 |

Then store the list of model CSVs in a directory "path/to/model/predictions"

from imagenet_x import get_factor_accuracies, error_ratio

factor_accs = get_factor_accuracies("path/to/model/predictions")
error_ratios = error_ratio(factor_accs)

This gives the following table

Error ratios

| model | pose | background | pattern | color | smaller | shape | partial_view | subcategory | texture | larger | darker | object_blocking | person_blocking | style | brighter | multiple_objects | |:---------|---------:|-------------:|----------:|--------:|----------:|--------:|---------------:|--------------:|----------:|---------:|---------:|------------------:|------------------:|--------:|-----------:|-------------------:| | DINO | 0.726197 | 1.06799 | 0.927478 | 1.1779 | 1.54369 | 1.64228 | 1.12906 | 1.95486 | 2.15032 | 1.24805 | 1.46777 | 1.93051 | 2.03486 | 1.70361 | 0.924938 | 1.4244 | | ResNet50 | 0.694739 | 1.1417 | 0.771442 | 1.18883 | 1.49743 | 1.74423 | 1.13236 | 2.10548 | 2.39386 | 1.31592 | 1.71502 | 1.92327 | 2.17128 | 1.92798 | 1.16639 | 1.97389 | | SimCLR | 0.774029 | 1.0867 | 0.911171 | 1.11955 | 1.54176 | 1.47304 | 1.01247 | 1.61814 | 2.0584 | 1.0121 | 1.12238 | 1.75552 | 2.03412 | 1.17686 | 0.879497 | 1.1907 | | ViT | 0.74067 | 1.09097 | 0.862456 | 1.15164 | 1.64401 | 1.53992 | 0.917235 | 2.01538 | 2.04465 | 1.29087 | 1.83403 | 1.98596 | 1.93631 | 1.60108 | 0.782348 |

We also provide some plotting utilities

from imagenet_x import plots

plots.model_comparison(factor_accs.reset_index(), fname="/path/to/save/fig.pdf|png")

Finally, we also provide a ImagenetX pytorch dataset that loads the imagenet samples and appends the factors of validation as a one hot encoded vector of 16 elements.

from imagenet_x.evaluate import ImageNetX, get_vanilla_transform

# Declare dataset
imagenet_val_path = '/path/to/imagenet'
transforms = get_vanilla_transform()
dataset = ImageNetX(imagenet_val_path, transform=transforms)

See this notebook to run the previous commands and for a sample evaluation loop on a resnet-18.

Paper results

To reproduce plots for models in the paper

You need python>=3.8 for plots and evaluation to work

python -m imagenet_x plots [--use-tex]

Generate aggregate results from model predictions

python -m imagenet_x aggregate --model-dirs path/to/model/predictions 

License

License file in root of directory.