🎉News! If you feel that your DAOD research has hit a bottleneck, welcome to check out our latest work on Adaptive Open-set Object Detection, which extends the target domain to the open set!

Three branches of the project:

Main branch (SIGMA): git clone https://github.com/CityU-AIM-Group/SIGMA.git
SIGMA++ branch: git clone -b SIGMA++ https://github.com/CityU-AIM-Group/SIGMA.git
FRCNN-SIGMA++ branch: git clone -b FRCNN-SIGMA++ https://github.com/CityU-AIM-Group/SIGMA.git

The features of SIGMA++:

More datasets and instructions
More stable and better results
From graph to hypergraph

SIGMA++ has found its final home now, indicating the end of this series of works. The growth of SIGMA++ is full of frustration: 👶 ➡ 🧒.

SCAN ➡ SCAN++ ➡ SIGMA ➡ SIGMA++

The main idea of the series of works: Model fine-grained feature points with graphs. We sincerely appreciate for all the readers showing interest in our works.

Honestly, due to the limited personal ability, our works still have many limitations, e.g., sub-optimal and redundant designs. Please forgive me. Nevertheless, we hope our works can inspire lots of good idea.

Best regards,
Wuyang Li
E-mail: wuyangli2-c@my.cityu.edu.hk

💡 Preparation

Installation

Check INSTALL.md for installation instructions. If you have any problem, feel free to screenshot your issue for me. Thanks.

Data preparation

More detailed preparation instructions are available here.

Step 1: Prepare benchmark datasets.

We follow EPM to construct the training and testing set by three following settings. Annotation files are available at onedrive.

Cityscapes -> Foggy Cityscapes

Download Cityscapes and Foggy Cityscapes dataset from the link. Particularly, we use leftImg8bit_trainvaltest.zip for Cityscapes and leftImg8bit_trainvaltest_foggy.zip for Foggy Cityscapes.
Download and extract the converted annotation from the following links: Cityscapes and Foggy Cityscapes (COCO format)

Extract the training sets from leftImg8bit_trainvaltest.zip, then move the folder leftImg8bit/train/ to Cityscapes/leftImg8bit/ directory.
Extract the training and validation set from leftImg8bit_trainvaltest_foggy.zip, then move the folder leftImg8bit_foggy/train/ and leftImg8bit_foggy/val/ to Cityscapes/leftImg8bit_foggy/ directory.

Sim10k -> Cityscapes (class car only)

Download Sim10k dataset and Cityscapes dataset from the following links: Sim10k and Cityscapes. Particularly, we use repro_10k_images.tgz and repro_10k_annotations.tgz for Sim10k and leftImg8bit_trainvaltest.zip for Cityscapes.
Download and extract the converted annotation from the following links: Sim10k (VOC format) and Cityscapes (COCO format)

Extract the training set from repro_10k_images.tgz and repro_10k_annotations.tgz, then move all images under VOC2012/JPEGImages/ to Sim10k/JPEGImages/ directory and move all annotations under VOC2012/Annotations/ to Sim10k/Annotations/.
Extract the training and validation set from leftImg8bit_trainvaltest.zip, then move the folder leftImg8bit/train/ and leftImg8bit/val/ to Cityscapes/leftImg8bit/ directory.

KITTI -> Cityscapes (class car only)

Download KITTI dataset and Cityscapes dataset from the following links: KITTI and Cityscapes. Particularly, we use data_object_image_2.zip for KITTI and leftImg8bit_trainvaltest.zip for Cityscapes.
Download and extract the converted annotation from the following links: KITTI (VOC format) and Cityscapes (COCO format).
Extract the training set from data_object_image_2.zip, then move all images under training/image_2/ to KITTI/JPEGImages/ directory.
Extract the training and validation set from leftImg8bit_trainvaltest.zip, then move the folder leftImg8bit/train/ and leftImg8bit/val/ to Cityscapes/leftImg8bit/ directory.

[DATASET_PATH]
└─ Cityscapes
   └─ cocoAnnotations
   └─ leftImg8bit
      └─ train
      └─ val
   └─ leftImg8bit_foggy
      └─ train
      └─ val
└─ KITTI
   └─ Annotations
   └─ ImageSets
   └─ JPEGImages
└─ Sim10k
   └─ Annotations
   └─ ImageSets
   └─ JPEGImages

Step 2: change the data root for your dataset at paths_catalog.py.

DATA_DIR = [$Your dataset root]

📦 Well-trained models

The ImageNet pretrained VGG-16 backbone (w/o BN) is available at link. You can use it if you cannot download the model through the link in the config file.
The well-trained models are available at this link).

We can get higher results than the reported ones with tailor-tuned hyperparameters.
E2E indicates end-to-end training for better reproducibility. Our config files are used for end-to-end training.
Two-stage/ longer training and turning learning rate will make the results more stable and get higer mAP/AP75.
After correcting a default hyper-parameter (as explained in the config file), Sim10k to City achieves better results than the reported ones.
You can set MODEL.MIDDLE_HEAD.GM.WITH_CLUSTER_UPDATE False to accelerate training greatly with ignorable performance drops. You'd better also make this change for bs=2 since we found it more friendly for the small batch-size training.
Results will be stable after the learning rate decline (in the training schedule).

| Source| Target| E2E|Metric | Backbone | mAP | AP@50 | AP@75 | file | | :-----:|:----:|:----: | :----: | :----:| :----: |:-----:| :----: | :----: | | City |Foggy | |COCO |V-16|24.0 |43.6|23.8|city_to_foggy_vgg16_43.58_mAP.pth| | City |Foggy | |COCO |V-16| 24.3 |43.9|22.6| city_to_foggy_vgg16_43.90_mAP.pth| | City |Foggy |$\checkmark$ |COCO |V-16| 22.0 |43.5|21.8| reproduced| | City |Foggy | |COCO |R-50| 22.7 |44.3|21.2| city_to_foggy_res50_44.26_mAP.pth| | City | BDD100k| |COCO|V-16 | - |32.7|- |city_to_bdd100k_vgg16_32.65_mAP.pth| | Sim10k| City | |COCO|V-16 | 33.4 |57.1 |33.8 |sim10k_to_city_vgg16_53.73_mAP.pth| | Sim10k |City |$\checkmark$ |COCO |V-16| 32.1 |55.2|32.1| reproduced| | KITTI | City | |COCO|V-16 | 22.6 |46.6 |20.0 |kitti_to_city_vgg16_46.45_mAP.pth|

🔥 Get start

NOTE: In the code comments, there is a small correction about batchsize: IMS_PER_BATACH=4 indicates 4 images per domain.

Train the model from the scratch with the default setting (batchsize = 4):

python tools/train_net_da.py \
        --config-file configs/SIGMA/xxx.yaml \

Test the well-trained model:

python tools/test_net.py \
        --config-file configs/SIGMA/xxx.yaml \
        MODEL.WEIGHT well_trained_models/xxx.pth

For example: test cityscapes to foggy cityscapes with VGG16 backbone.

python tools/test_net.py \
         --config-file configs/SIGMA/sigma_vgg16_cityscapace_to_foggy.yaml \
         MODEL.WEIGHT well_trained_models/city_to_foggy_vgg16_43.58_mAP.pth

✨ Quick Tutorials

See sigma_vgg16_cityscapace_to_foggy.yaml to understand APIs.
We modify the trainer to meet the requirements of SIGMA.
GM is integrated in this "middle layer": [graph_matching_head](https://github.com/CityU-AIM-Group/SIGMA/blob/main/fcos_core/modeling/rpn/fcos/graph_matching_hea

SIGMA

Install / Use

README

SIGMA: Semantic-complete Graph Matching For Domain Adaptive Object Detection (CVPR-22 ORAL)