<div align="center"> <h1> Geometry-guided Kernel Transformer </h1> <span><font size="5", > Efficient and Robust 2D-to-BEV Representation Learning via Geometry-guided Kernel Transformer </font></span> </br> Shaoyu Chen*, Tianheng Cheng*, <a href="https://xinggangw.info/">Xinggang Wang</a><sup><span>†</span></sup>, Wenming Meng, <a href="https://scholar.google.com/citations?user=pCY-bikAAAAJ&hl=zh-CN">Qian Zhang</a>, <a href="http://eic.hust.edu.cn/professor/liuwenyu/"> Wenyu Liu</a>

(<span>*</span>: equal contribution, <span>†</span>: corresponding author) <br>

<div><a href="https://arxiv.org/pdf/2206.04584.pdf">[arXiv Preprint]</a></div> </div>

News

October 14, 2022: We've released code & models for map-view segmentation
June 9, 2022: We've released the tech report for Geometry-guided Kernel Transformer (GKT). This work is still in progress and code/models are coming sonn. Please stay tuned! ☕️

Introduction

Framework

We present a novel and efficient 2D-to-BEV transformation, Geometry-guided Kernel Transformer (GKT).

GKT leverages geometric priors to guide the transformers to focus on discriminative regions for generating BEV representation with surrouding-view image features.
GKT is based on kernel-wise attention and much efficient, especially with LUT indexing.
GKT is robust to the deviation of cameras, making the 2D-to-BEV transformation more stable and reliable.

Getting Started

git clone https://github.com/hustvl/GKT.git

Map-view nuScenes Segmentation

Models

| Method | Kernel | mIoU (Setting 1) | mIoU (Setting 2) | FPS | model | | :----: | :----: | :--------------: | :--------------: | :-: | :---: | | CVT | - | 39.3 | 37.2 | 34.1 | model | | GKT | 7x1 | 41.4 | 38.0 | 45.6 | model |

Note: FPS are measured on one 2080 Ti GPU.

Usage

For map-view nuScenes segmentation, we mainly build the GKT based on the awesome CrossViewTransformer.

# map-view segmentation
cd segmentation

Prerequisites

# install dependencies
pip install -r reuqirements.txt
pip install -e .

Preparing the Dataset

Training / Testing / Benchmarking

Pretrained model

Download the pretrained model efficientnet-b4-6ed6700e.pth

mkdir pretrained_models
cd pretrained_models
# place the pretrained model here

Training

python scripts/train.py +experiment=gkt_nuscenes_vehicle_kernel_7x1.yaml  data.dataset_dir=<path/to/nuScenes> data.labels_dir=<path/to/labels>

Testing

Using the absolute path of the checkpoint is better.

python scripts/eval.py +experiment=gkt_nuscenes_vehicle_kernel_7x1.yaml data.dataset_dir=<path/to/nuScenes> data.labels_dir=<path/to/labels> experiment.ckptt <path/to/checkpoint>

Evalutating Speed

python scripts/speed.py +experiment=gkt_nuscenes_vehicle_kernel_7x1.yaml data.dataset_dir=<path/to/nuScenes> data.labels_dir=<path/to/labels>

3D Object Detection

coming soon.

Acknowledgements

We sincerely appreciate the awesome repos cross_view_transformers and fiery!

License

GKT is released under the MIT Licence.

Citation

If you find GKT is useful in your research or applications, please consider giving us a star 🌟 and citing it by the following BibTeX entry.

@article{GeokernelTransformer,
  title={Efficient and Robust 2D-to-BEV Representation Learning via Geometry-guided Kernel Transformer},
  author={Chen, Shaoyu and Cheng, Tianheng and Wang, Xinggang and Meng, Wenming and Zhang, Qian and Liu, Wenyu},
  journal={arXiv preprint arXiv:2206.04584},
  year={2022}
}

GKT

Install / Use

README