Universal Representation Learning and Task-specific Adaptation for Few-shot Learning

A universal representation learning algorithm that learns a set of well-generalized representations via a single universal network from multiple diverse visual datasets and task-specific adaptation techniques for few-shot learning.

Universal Representation Learning from Multiple Domains for Few-shot Classification,
Wei-Hong Li, Xialei Liu, Hakan Bilen,
ICCV 2021 (arXiv 2103.13841)

Cross-domain Few-shot Learning with Task-specific Adapters,
Wei-Hong Li, Xialei Liu, Hakan Bilen,
CVPR 2022 (arXiv 2107.00358)

Universal Representations: A Unified Look at Multiple Task and Domain Learning,
Wei-Hong Li, Xialei Liu, Hakan Bilen,
IJCV 2023 (arXiv 2204.02744)

Updates

November'22, Code of different options for task-specific adapters is released! See TSA.
March'22, Code for Cross-domain Few-shot Learning with Task-specific Adapters (CVPR'22) is now available! See TSA.
Oct'21, Code and models for Universal Representation Learning from Multiple Domains for Few-shot Classification (ICCV'21) are now available!

Features at a glance

We train a single universal (task-agnostic) network on 8 visual (training) datasets on Meta-dataset: ImageNet, Omniglot, Aircraft, Birds, Textures, Quick Draw, Fungi, VGG Flower, with state-of-the-art performances on all (13) testing datasets for few-shot learning.
During meta-testing, the universal representations can be efficiently adapted by our proposed pre-classifier alignment (a linear transformation) learned on the support set to transform the representations to a more discriminative space.
We propose to attach a set of light weight task-specific adapters to the universal network (the universal network can be learned from multiple datasets or one single diverse dataset, e.g. ImageNet) and learn task-specific adapters on the support set from scratch for adapting the few-shot model to the tasks from unseen domains.
Our method also allows Low Rank Adaptation (LoRA), i.e. learning the attached adapters can be treating as learning two decomposed matrices, which is more efficient while maintains good performance.
We systematically study various combinations of several design choices for task-specific adaptation, which have not been explored before, including adapter connection types (serial or residual), parameterizations (matrix and its decomposed variations, channelwise operations) and estimation of task-specific parameters.
Attaching parameteric adapters in matrix form to convolutional layers with residual connections significantly boosts the state-of-the-art performance in most domains, especially resulting in superior performance in unseen domains on Meta-Dataset
In this repo, we provide code of the URL, the best task adaptation strategy in TSA, and other baselines like SDL, vanilla MDL and other evaluation settings.

Main results on Meta-dataset

Multi-domain setting (meta-train on 8 datasets and meta-test on 13 datasets).

Test Datasets |TSA (Ours) |URL (Ours) |MDL |Best SDL |tri-M [8] |FLUTE [7] |URT [6] |SUR [5] |Transductive CNAPS [4] |Simple CNAPS [3] |CNAPS [2]
---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|--------------------------- Avg rank |1.5 |2.7 |7.1 |6.7 |5.5 |5.1 |6.7 |6.9 |5.7 |7.2 |-
Avg Seen |80.2 |80.0 |76.9 |76.3 |74.5 |76.2 |76.7 |75.2 |75.1 |74.6 |71.6
Avg Unseen |77.2 |69.3 |61.7 |61.9 |69.9 |69.9 |62.4 |63.1 |66.5 |65.8 |-
Avg All |79.0 |75.9 |71.1 |70.8 |72.7 |73.8 |71.2 |70.5 |71.8 |71.2 |-

Single-domain setting (Meta-train on ImageNet and meta-test on 13 datasets).

Test Datasets |TSA-ResNet34 (Ours) |TSA-ResNet18 (Ours) |CTX-ResNet34 [10] |ProtoNet-ResNet34 [10] |FLUTE [7] |BOHB [9] |ALFA+fo-Proto-MAML [1] |fo-Proto-MAML [1] |ProtoNet [1] |Finetune [1]
---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|--------------------------- Avg rank |1.5 |2.8 |1.8 |5.5 |8.9 |6.0 |5.3 |7.0 |8.3 |7.9
Avg Seen |63.7 |59.5 |62.8 |53.7 |46.9 |51.9 |52.8 |49.5 |50.5 |45.8
Avg Unseen |76.2 |71.9 |75.6 |61.1 |53.2 |60.0 |62.4 |58.4 |56.7 |58.2
Avg All |74.9 |70.7 |74.3 |60.4 |52.6 |59.2 |61.4 |57.5 |56.1 |57.0

Model Zoo

Single-domain networks (one for each dataset)
A single universal network (URL) learned from 8 training datasets
ImageNet model learned with ResNet34 and higher resolution images (224x224) like CTX [10], coming soon!

Dependencies

This code requires the following:

Python 3.6 or greater
PyTorch 1.0 or greater
TensorFlow 1.14 or greater

Installation

Clone or download this repository.
Configure Meta-Dataset:
- Follow the "User instructions" in the Meta-Dataset repository for "Installation" and "Downloading and converting datasets".
- Edit ./meta-dataset/data/reader.py in the meta-dataset repository to change dataset = dataset.batch(batch_size, drop_remainder=False) to dataset = dataset.batch(batch_size, drop_remainder=True). (The code can run with drop_remainder=False, but in our work, we drop the remainder such that we will not use very small batch for some domains and we recommend to drop the remainder for reproducing our methods.)
- To test unseen domain (out-of-domain) performance on additional datasets, i.e. MNIST, CIFAR-10 and CIFAR-100, follow the installation instruction in the CNAPs repository to get these datasets.

Initialization

Before doing anything, first run the following commands.

ulimit -n 50000
export META_DATASET_ROOT=<root directory of the cloned or downloaded Meta-Dataset repository>
export RECORDS=<the directory where tf-records of MetaDataset are stored>

Note the above commands need to be run every time you open a new command shell.

Enter the root directory of this project, i.e. the directory where this project was cloned or downloaded.

Universal Representation Learning from Multiple Domains for Few-shot Classification

<img src="./figures/universal.png" style="width:60%"> Figure 1. URL - Universal Representation Learning.

Train the Universal Representation Learning Network

The easiest way is to download our [pre-trained URL model](https://drive.google.com/file/d/1Dv8TX6iQ

URL

Install / Use

README