ImagenetX : Understanding model mistakes with factors of variation annotations

Code to load annotations, evaluate models and reproduce paper plots. See [paper], [website], [colab]

Installation

If you just want to load the annotations :

pip install imagenet-x

Other installations:

• Install from repo clone : pip install -e .
• Reproduce plots from the paper and use dataset loader: pip install 'imagenet-x[all]'

Usage

To load the annotations

from imagenet_x import load_annotations

annotations = load_annotations()

This will output the following table

See this notebook for some sample images and annotations

One can also directly download the raw annotation files stored in the annotations folder. There are 4 json line files imagenet_x_[train|val]_[multi|top]_factor.jsonl that have entries such as the following:

{"file_name":"ILSVRC2012_val_00004487.JPEG","class":762,"multiple_objects":0,"background":1,"color":0,"brighter":0,"darker":0,"style":0,"larger":1,"smaller":0,"object_blocking":0,"person_blocking":0,"partial_view":0,"pattern":0,"pose":1,"shape":0,"subcategory":0,"texture":0,"justification":"close up of a pan fried sea bass. ","one_word":"sea bass close up"}

To generate plots for a new model

Generate the predictions of your model on the Imagenet Validation set in a csv file with 3 columns

Then store the list of model CSVs in a directory "path/to/model/predictions"

from imagenet_x import get_factor_accuracies, error_ratio

factor_accs = get_factor_accuracies("path/to/model/predictions")
error_ratios = error_ratio(factor_accs)

This gives the following table

Error ratios

We also provide some plotting utilities

from imagenet_x import plots

plots.model_comparison(factor_accs.reset_index(), fname="/path/to/save/fig.pdf|png")

Finally, we also provide a ImagenetX pytorch dataset that loads the imagenet samples and appends the factors of validation as a one hot encoded vector of 16 elements.

from imagenet_x.evaluate import ImageNetX, get_vanilla_transform

# Declare dataset
imagenet_val_path = '/path/to/imagenet'
transforms = get_vanilla_transform()
dataset = ImageNetX(imagenet_val_path, transform=transforms)

See this notebook to run the previous commands and for a sample evaluation loop on a resnet-18.

Paper results

To reproduce plots for models in the paper

You need python>=3.8 for plots and evaluation to work

python -m imagenet_x plots [--use-tex]

Generate aggregate results from model predictions

python -m imagenet_x aggregate --model-dirs path/to/model/predictions 

License

License file in root of directory.