Data-Efficient Deep Learning Benchmark (DEIC)

This repository contains descriptions, download instructions, and canonical train/validation/test splits for the six datasets used as a benchmark for data-efficient image classification in the following paper:

Lorenzo Brigato, Björn Barz, Luca Iocchi, and Joachim Denzler.
Tune It or Don't Use It: Benchmarking Data-Efficient Image Classification.
ICCV 2021 Workshop on Visual Inductive Priors for Data-Efficient Deep Learning (VIPriors).

:warning: For the newest version of the benchmark, and the full PyTorch implementations of several data-efficient methods, consider visiting:

https://github.com/lorenzobrigato/gem

Datasets

| Dataset | Classes | Imgs/Class | Trainval | Test | Problem Domain | Data Type | |:---------------|--------:|-----------:|---------:|-------:|:---------------|:--------------| | ImageNet | 1,000 | 50 | 50,000 | 50,000 | Natural Images | RGB | | ciFAIR-10 | 10 | 50 | 500 | 10,000 | Natural Images | RGB (32x32) | | CUB | 200 | 30 | 5,994 | 5,794 | Fine-Grained | RGB | | EuroSAT | 10 | 50 | 500 | 19,500 | Remote Sensing | Multispectral | | ISIC 2018 | 7 | 80 | 560 | 1,944 | Medical | RGB | | CLaMM | 12 | 50 | 600 | 2,000 | Handwriting | Grayscale |

For all datasets except CUB, we use subsampled versions of the original datasets with fewer images per class.

Canonical Splits

The datasets directory contains one sub-directory for each dataset in our benchmark. These directories contain the files train.txt, val.txt, and test.txt specifying the three subsets. Additionally, the file trainval.txt is simply the concatenation of train.txt and val.txt. This subset can be used for the final training before evaluating a method on the test set. Development and hyper-parameter optimization, however, should only be conducted using the training and validation sets.

The aforementioned files list all images contained in the respective subset, one per line, along with their class labels. Each line contains the filename of an image followed by a space and the numeric index of its label.

The only exception from this common format is ciFAIR-10, since it does not have filenames. A description of the split can be found in the README.md of the respective directory.