<div align="center"> <img src="images/cvlab.png" align="right" width=8%>

DEVIANT: Depth EquiVarIAnt NeTwork for Monocular 3D Object Detection

KITTI Demo | KITTI Eqv Error Demo | Waymo Demo | Project | Talk | Slides | Poster

<p style="font-size:1.2em"> <a href="https://sites.google.com/view/abhinavkumar"><strong>Abhinav Kumar</strong></a>¹ · <a href="https://garrickbrazil.com"><strong>Garrick Brazil</strong></a>² · <a href="https://www.linkedin.com/in/enrique-corona-0752b84"><strong>Enrique Corona</strong></a>³ · <a href="https://www.linkedin.com/in/parchami"><strong>Armin Parchami</strong></a>³ · <a href="http://www.cse.msu.edu/~liuxm/index2.html"><strong>Xiaoming Liu</strong></a>¹<br> ¹Michigan State University, ²FAIR, Meta AI, ³Ford Motor Company </p>

in ECCV 2022

<p align="center"> <img src="images/waymo_detection_demo.gif" width="512"> </p> <p align="center"> <img src="images/idea_overview.png"> </p> </div>

Modern neural networks use building blocks such as convolutions that are equivariant to arbitrary 2D translations $(t_u, t_v)$. However, these vanilla blocks are not equivariant to arbitrary 3D translations $(t_x, t_y, t_z)$ in the projective manifold. Even then, all monocular 3D detectors use vanilla blocks to obtain the 3D coordinates, a task for which the vanilla blocks are not designed for. This paper takes the first step towards convolutions equivariant to arbitrary 3D translations in the projective manifold. Since the depth is the hardest to estimate for monocular detection, this paper proposes Depth EquiVarIAnt NeTwork (DEVIANT) built with existing scale equivariant steerable blocks. As a result, DEVIANT is equivariant to the depth translations $(t_z)$ in the projective manifold whereas vanilla networks are not. The additional depth equivariance forces the DEVIANT to learn consistent depth estimates, and therefore, DEVIANT achieves state-of-the-art monocular 3D detection results on KITTI and Waymo datasets in the image-only category and performs competitively to methods using extra information. Moreover, DEVIANT works better than vanilla networks in cross-dataset evaluation.

Much of the codebase is based on GUP Net. Some implementations are from GrooMeD-NMS and PCT. Scale Equivariant Steerable (SES) implementations are from SiamSE.

Citation

If you find our work useful in your research, please consider starring the repo and citing:

@inproceedings{kumar2022deviant,
   title={{DEVIANT: Depth EquiVarIAnt NeTwork for Monocular $3$D Object Detection}},
   author={Kumar, Abhinav and Brazil, Garrick and Corona, Enrique and Parchami, Armin and Liu, Xiaoming},
   booktitle={ECCV},
   year={2022}
}

Setup

• Requirements

  1. Python 3.7
  2. PyTorch 1.10
  3. Torchvision 0.11
  4. Cuda 11.3
  5. Ubuntu 18.04/Debian 8.9

This is tested with NVIDIA A100 GPU. Other platforms have not been tested. Clone the repo first. Unless otherwise stated, the below scripts and instructions assume the working directory is the directory DEVIANT:

git clone https://github.com/abhi1kumar/DEVIANT.git
cd DEVIANT

• Cuda & Python

Build the DEVIANT environment by installing the requirements:

conda create --name DEVIANT --file conda_GUP_environment_a100.txt
conda activate DEVIANT
pip install opencv-python pandas

• KITTI, nuScenes and Waymo Data

Follow instructions of data_setup_README.md to setup KITTI, nuScenes and Waymo as follows:

DEVIANT
├── data
│      ├── KITTI
│      │      ├── ImageSets
│      │      ├── kitti_split1
│      │      ├── training
│      │      │     ├── calib
│      │      │     ├── image_2
│      │      │     └── label_2
│      │      │
│      │      └── testing
│      │            ├── calib
│      │            └── image_2
│      │
│      ├── nusc_kitti
│      │      ├── ImageSets
│      │      ├── training
│      │      │     ├── calib
│      │      │     ├── image
│      │      │     └── label
│      │      │
│      │      └── validation
│      │            ├── calib
│      │            ├── image
│      │            └── label
│      │
│      └── waymo
│             ├── ImageSets
│             ├── training
│             │     ├── calib
│             │     ├── image
│             │     └── label
│             │
│             └── validation
│                   ├── calib
│                   ├── image
│                   └── label
│
├── experiments
├── images
├── lib
├── nuscenes-devkit        
│ ...

• AP Evaluation

Run the following to generate the KITTI binaries corresponding to R40:

sudo apt-get install libopenblas-dev libboost-dev libboost-all-dev gfortran
sh data/KITTI/kitti_split1/devkit/cpp/build.sh

We finally setup the Waymo evaluation. The Waymo evaluation is setup in a different environment py36_waymo_tf to avoid package conflicts with our DEVIANT environment:

# Set up environment
conda create -n py36_waymo_tf python=3.7
conda activate py36_waymo_tf
conda install cudatoolkit=11.3 -c pytorch

# Newer versions of tf are not in conda. tf>=2.4.0 is compatible with conda.
pip install tensorflow-gpu==2.4
conda install pandas
pip3 install waymo-open-dataset-tf-2-4-0 --user

To verify that your Waymo evaluation is working correctly, pass the ground truth labels as predictions for a sanity check. Type the following:

/mnt/home/kumarab6/anaconda3/envs/py36_waymo_tf/bin/python -u data/waymo/waymo_eval.py --sanity

You should see AP numbers as 100 in every entry after running this sanity check.

Training

Train the model:

chmod +x scripts_training.sh
./scripts_training.sh

The current Waymo config files use the full val set in training. For Waymo models, we had subsampled Waymo validation set by a factor of 10 (4k images) to save training time as in DD3D. Change val_split_name from 'val' to 'val_small' in waymo configs to use subsampled Waymo val set.

Testing Pre-trained Models

Model Zoo

We provide logs/models/predictions for the main experiments on KITTI Val /KITTI Test/Waymo Val data splits available to download here.

| Data_Splits | Method | Config<br/>(Run) | Weight<br>/Pred | Metrics | All<br/>(0.7) | Easy<br/>(0.7) | Med<br/>(0.7) | Hard<br/>(0.7) |All<br/>(0.5) | Easy<br/>(0.5) | Med<br/>(0.5) | Hard<br/>(0.5) | |------------|---------|------------------------------------------------------------------|----------|--------|----------|-----------|----------|-----------|----------------|----|--------------|-----------| | KITTI Val | GUP Net | run_201 | gdrive | AP40 | - | 21.10 | 15.48 | 12.88 | - | 58.95 | 43.99 | 38.07 | KITTI Val | DEVIANT | run_221 | gdrive | AP40 | - | 24.63 | 16.54 | 14.52 | - | 61.00 | 46.00 | 40.18 | KITTI Test | DEVIANT | run_250 | gdrive | AP40 | - | 21.88 | 14.46 | 11.89 | - | - | - | -
| Waymo Val | GUP Net | run_1050 | gdrive | APH-L1| 2.27 | 6.11 | 0.80 | 0.03 | 9.94 | 24.59 | 4.78 | 0.22 | Waymo Val | DEVIANT | run_1051 | gdrive | APH-L1| 2.67 | 6.90 | 0.98 | 0.02 | 10.89 | 26.64 | 5.08 | 0.18

Testing

Make output folder in the DEVIANT directory:

mkdir output

Place models in the output folder as follows:

DEVIANT