Understanding Few-Shot Learning: Measuring Task Relatedness and Adaptation Difficulty via Attributes

This repository is the official implementation of the paper "Understanding Few-Shot Learning: Measuring Task Relatedness and Adaptation Difficulty via Attributes" in Neural Information Processing Systems (NeurIPS 2023). In this project, we provide the Task Attribute Distance (TAD) metric to quantify the task relatedness and measure the adaptation difficulty of novel tasks.

Dependenices

The code is built with following libraries:

• python 3.7
• PyTorch 1.7.1
• cv2
• matplotlib
• sklearn
• tensorboard
• h5py
• tqdm

Installation

conda create -n TAD python=3.7
source activate TAD
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=11.0 -c pytorch
pip install -r requirements.txt

Dataset prepare

Please download the CUB and SUN datasets, then put them under the path of filelists/<dataset name>/.

Here we provide a link of CUB dataset and related files.

Training

To train the FSL models (such as ProtoNet) on CUB dataset, run this command:

bash scripts/train/cub_protonet.sh

Evaluation

To evaluate models on CUB, run:

bash scripts/test/cub_protonet.sh

Plot task distance and accuracy

To estimate the average TAD between each novel task and training tasks, then plot a figure of average TAD and accuracy, run:

bash scripts/test/plot_distance_acc.sh

Fast start

Here we provide some pretrained models for fast start.

• ProtoNet (Conv4NP) trained on CUB in the 5-way 1-shot setting

Download the pretrained model at file path checkpoints/CUB/Conv4NP_protonet_0_aug_5way_1shot/, and then run the command in Plot task distance and accuracy part.

Our codebase is developed based on the baseline++ from the paper A Closer Look at Few-shot Classification and COMET from the paper Concept Learners for Few-Shot Learning.