This repository is no longer maintained. I am no longer actively maintaining iCAN. Please refer to our ECCV 2020 work DRG for a stronger HOI detection framework in PyTorch.

iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection

Official TensorFlow implementation for iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection.

See the project page for more details. Please contact Chen Gao (chengao@vt.edu) if you have any questions.

<img src='misc/HOI.gif'>

Prerequisites

This codebase was developed and tested with Python2.7, Tensorflow 1.1.0 or 1.2.0, CUDA 8.0 and Ubuntu 16.04.

Installation

1. Clone the repository.

   git clone https://github.com/vt-vl-lab/iCAN.git
   

2. Download V-COCO and HICO-DET dataset. Setup V-COCO and COCO API. Setup HICO-DET evaluation code.

   chmod +x ./misc/download_dataset.sh 
   ./misc/download_dataset.sh 
   
   # Assume you cloned the repository to `iCAN_DIR'.
   # If you have downloaded V-COCO or HICO-DET dataset somewhere else, you can create a symlink
   # ln -s /path/to/your/v-coco/folder Data/
   # ln -s /path/to/your/hico-det/folder Data/
   

Evaluate V-COCO and HICO-DET detection results

1. Download detection results

   chmod +x ./misc/download_detection_results.sh 
   ./misc/download_detection_results.sh
   

2. Evaluate V-COCO detection results using iCAN

   python tools/Diagnose_VCOCO.py eval Results/300000_iCAN_ResNet50_VCOCO.pkl
   

3. Evaluate V-COCO detection results using iCAN (Early fusion)

   python tools/Diagnose_VCOCO.py eval Results/300000_iCAN_ResNet50_VCOCO_Early.pkl
   

4. Evaluate HICO-DET detection results using iCAN

   cd Data/ho-rcnn
   matlab -r "Generate_detection; quit"
   cd ../../
   

   Here we evaluate our best detection results under Results/HICO_DET/1800000_iCAN_ResNet50_HICO. If you want to evaluate a different detection result, please specify the filename in Data/ho-rcnn/Generate_detection.m accordingly.

Error diagnose on V-COCO

1. Diagnose V-COCO detection results using iCAN

   python tools/Diagnose_VCOCO.py diagnose Results/300000_iCAN_ResNet50_VCOCO.pkl
   

2. Diagnose V-COCO detection results using iCAN (Early fusion)

   python tools/Diagnose_VCOCO.py diagnose Results/300000_iCAN_ResNet50_VCOCO_Early.pkl
   

Training

1. Download COCO pre-trained weights and training data

   chmod +x ./misc/download_training_data.sh 
   ./misc/download_training_data.sh
   

2. Train an iCAN on V-COCO

   python tools/Train_ResNet_VCOCO.py --model iCAN_ResNet50_VCOCO --num_iteration 300000
   

3. Train an iCAN (Early fusion) on V-COCO

   python tools/Train_ResNet_VCOCO.py --model iCAN_ResNet50_VCOCO_Early --num_iteration 300000
   

4. Train an iCAN on HICO-DET

   python tools/Train_ResNet_HICO.py --num_iteration 1800000
   

Testing

1. Test an iCAN on V-COCO

    python tools/Test_ResNet_VCOCO.py --model iCAN_ResNet50_VCOCO --num_iteration 300000
   

2. Test an iCAN (Early fusion) on V-COCO

    python tools/Test_ResNet_VCOCO.py --model iCAN_ResNet50_VCOCO_Early --num_iteration 300000
   

3. Test an iCAN on HICO-DET

   python tools/Test_ResNet_HICO.py --num_iteration 1800000
   

Visualizing V-COCO detections

Check tools/Visualization.ipynb to see how to visualize the detection results.

Demo/Test on your own images

0. To get the best performance, we use Detectron as our object detector. For a simple demo purpose, we use tf-faster-rcnn in this section instead.
1. Clone and setup the tf-faster-rcnn repository.

   cd $iCAN_DIR
   chmod +x ./misc/setup_demo.sh 
   ./misc/setup_demo.sh
   

2. Put your own images to demo/ folder.
3. Detect all objects

   # images are saved in $iCAN_DIR/demo/
   python ../tf-faster-rcnn/tools/Object_Detector.py --img_dir demo/ --img_format png --Demo_RCNN demo/Object_Detection.pkl
   

4. Detect all HOIs

   python tools/Demo.py --img_dir demo/ --Demo_RCNN demo/Object_Detection.pkl --HOI_Detection demo/HOI_Detection.pkl
   

5. Check tools/Demo.ipynb to visualize the detection results.

Citation

If you find this code useful for your research, please consider citing the following papers:

@inproceedings{gao2018ican,
author    = {Gao, Chen and Zou, Yuliang and Huang, Jia-Bin}, 
title     = {iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection}, 
booktitle = {British Machine Vision Conference},
year      = {2018}
}

Acknowledgement

Codes are built upon tf-faster-rcnn. We thank Jinwoo Choi for the code review.