Key-point Detection based Online Real-Time Spatio-Temporal Action Localization

Kalana Abeywardena, Sakuna Jayasundara, Sachira Karunasena, Shechem Sumanthiran, Dr. Peshala Jayasekara, Dr. Ranga Rodrigo

<p align='justify'> Real-time and online action localization in a video is a critical yet highly challenging problem. Accurate action localization requires utilization of both temporal and spatial information. Recent attempts achieve this by using computationally intensive 3D CNN architectures or highly redundant two-stream architectures with optical flow, making them both unsuitable for real-time, online applications. To accomplish activity localization under highly challenging real-time constraints, we propose utilizing fast and efficient key-point based bounding box prediction to spatially localize actions. We then introduce a tube-linking algorithm that maintains the continuity of action tubes temporally in the presence of occlusions. Further, we eliminate the need for a two-stream architecture by combining temporal and spatial information into a cascaded input to a single network, allowing the network to learn from both types of information. Temporal information is efficiently extracted using a structural similarity index map as opposed to computationally intensive optical flow. Despite the simplicity of our approach, our lightweight end-to-end architecture achieves state-of-the-art frame-mAP on the challenging UCF101-24 dataset, demonstrating a performance gain of 6.4% over the previous best online methods. We also achieve state-of-the-art video-mAP results compared to both online and offline methods. Moreover, our model achieves a frame rate of 41.8 FPS, which is a 10.7% improvement over contemporary real-time methods. </p> <p align="center"> <img src="figures/NewArchitecture.png"> <em>Proposed Architecture</em> </p>

Highlights

• Utilize key-point-based detection architecture for the first time for the task of ST action localization, which reduces model complexity and inference time over traditional anchor-box-based approaches.
• Demonstrate that the explicit computation of OF is unnecessary and that the SSIM index map obtains sufficient inter-frame temporal information.
• A single network provided with both spatial and temporal information, and allowing it to extract necessary information through discriminative learning.
• An efficient tube-linking algorithm that extrapolates the tubes for a short period using past detections for real-time deployment.

Table of Content

1. Installation
2. Datasets
3. Training CenterNet
4. Saving Detections
5. Online Tube Generation
6. Performance
7. Citation
8. Reference

Installation

The code was tested on Ubuntu 18.04, with Anaconda Python 3.7 and PyTorch v1.4.0. NVIDIA GPUs are needed for both training and testing. After install Anaconda:

0. [Optional but recommended] create a new conda environment and activate the environment.

conda create --name CenterNet python=3.7
conda activate CenterNet

1. Install pytorch 1.4.0:

conda install pytorch=1.4.0 torchvision -c pytorch

Based on original repository, there can be slight reduction in performances for spatial localization with cudann batch normalization enabled. You can manually open torch/nn/functional.py and find the line with torch.batch_norm and replace the torch.backends.cudnn.enabled with False.

2. Install COCOAPI:

# COCOAPI=/path/to/clone/cocoapi
git clone https://github.com/cocodataset/cocoapi.git $COCOAPI
cd $COCOAPI/PythonAPI
make
python setup.py install --user

3. Install the requirements:

pip install -r requirements.txt

Datasets

<p align='justify'> We evaluate our framework on two datasets, <a href=https://www.crcv.ucf.edu/data/UCF101.php>UCF101-24</a> and <a href=http://jhmdb.is.tue.mpg.de/>J-HMDB21</a>. UCF101-24 is a subset of UCF101^[1] dataset with ST labels, having 3207 untrimmed videos with 24 action classes, that may contain multiple instances for the same action class. J-HMDB-21 is a subset of the HMDB51^[2] dataset having 928 temporally trimmed videos with 21 actions, each containing a single action instance.

Download the datasets and extract the frames. Place the extracted frames in <emp>rgb-images</emp> in the respective dataset directory in Datasets. The data directory should look as follows:

</p> <p align="center"> <img src="figures/sample directory tree.png"> <em>Sample directory tree for J-HMDB21</em> </p>

Training CenterNet

Setting up the CenterNet

When seting up CenterNet, we followed the instructions mentioned in their official repository. The modified scripts of CenterNet for Action Detection are provided in this repository.

To install DCNv2, follow the below instructions:

1. Build NMS

cd CenterNet\src\lib\external
#python setup.py install
python setup.py build_ext --inplace

Comment out the parameter in setup.py when building nms extension to solve invalid numeric argument /Wno-cpp: #extra_compile_args=["-Wno-cpp", "-Wno-unused-function"] (the provided script by us has made the changes already).

2. Clone and build original DCNv2

cd CenterNet\src\lib\models\networks
rm -rf DCNv2
git clone https://github.com/CharlesShang/DCNv2

After cloning the original DCNv2, navigate to the directory and make the following changes:

cd DCNv2
vim cuda/dcn_va_cuda.cu
"""
# extern THCState *state;
THCState *state = at::globalContext().lazyInitCUDA();
"""

Finally, execute the following command to build DCNv2:

python setup.py build develop

Training from the scratch

To train from the scratch with either UCF101-24 or J-HMDB21 datasets, the following command can be run.

python CUDA_VISIBLE_DEVICES=0 python main_SMDouble.py --dataset <dataset> --gpus <gpu id> --exp_id <save dir name> --task doubleSM --num_epochs <epochs (default: 60)> --variant <variation (default: 1)> 

Resuming from saved checkpoint

To resume the training from the last checkpoint saved, run the following command.

python CUDA_VISIBLE_DEVICES=0 python main_SMDouble.py --dataset <dataset> --gpus <gpu id> --exp_id <save dir name> --task doubleSM --num_epochs <epochs (default: 60)> --variant <variation (default: 1)> --resume 

Further, to resume the training from a specific chekpoint saved, run the following command.

python CUDA_VISIBLE_DEVICES=0 python main_SMDouble.py --dataset <dataset> --gpus <gpu id> --exp_id <save dir name> --task doubleSM --num_epochs <epochs (default: 60)> --variant <variation (default: 1)> --resume --load_model <path to the saved model>

Transfer Learning using the best checkpoint

To tranfer learn from a pre-trained checkpoint, run the following command.

python CUDA_VISIBLE_DEVICES=0 python main_SMDouble.py --dataset <dataset> --gpus <gpu id> --exp_id <save dir name> --task doubleSM --num_epochs <epochs (default: 60)> --variant <variation (default: 1)> --load_model /path/to/checkpoint

The pre-trained model checkpoints trained on J-HMDB21 and UCF101-24 datasets can be downloaded from here. Place the chekpoints at ./CenterNet/exp/$DATASET_NAME/dla34/rgb/$CHKPT_NAME to be compatible with the directory path definitions in the Centernet scripts.

Saving Detections

For evaluation, the spatial detections needs to be saved as <I>.mat </I> files. First, navigate to ./Save Detections/ and execute the following command:

python CUDA_VISIBLE_DEVICES=0 python SaveDetections.py --dataset <dataset> --ngpu <gpu id> --exp_id <save dir name> --task doubleSM --frame_gap <default:1> --variant <default:1> --load_model /path/to/checkpoint --result_root /path/to/detections

Online Tube Generation

After the spatial detections are saved for each video, the action tubes and paths are generated using the proposed online tube generation algorithm and its variation that are based on the original implementation which is also provided for comparison. The codes can be found in ./online-tubes/.

• To run the code, you will need to install MATLAB. You can install a free trial for testing purposes. Make sure you add the MATLAB installation path to the conda environment if you are executing scripts using command line.
• If you only have command line priviledges, you can install Octave and execute the tube generation.

Executing with MATLAB

1. Navigate to the respective directory:

cd ./online-tubes/EXP_without_BBOX

2. Change the paths based on where the data (saved detections) is located and results need saving in I01onlineTubes.m and utils/initDatasetOpts.m.
3. Execute I01onlineTubes.m. When executing using command line:

matlab -batch "I01onlineTubes.m"

Executing with Octave

1. Navigate to the respective directory:

cd ./online-tubes/EXP_without_BBOX

2. Change the paths based on where the data (saved detections) is located and results need saving in I01onlineTubes.m and utils/initDatasetOpts.m.
3. Execute I01onlineTubes.m. When executing using command line:

octave I01onlineTubes.m

There can be errors when running the current scripts in Octave. This is due to -v7.3 argument used in save() function in MATLAB scripts. You can simply remove the -v7.3 argument in save() functions and run without errors.

Performance

<p align='