OpenCC

Automatic driving long tail / corner cases datasets

Corner cases (CC) are data that occur infrequently or represent a critical situation and are only available in datasets to a limited extent, if at all. However, for machine learning (ML), CC are important as they are required for training, verification, and improved performance of ML models during inference within automated driving systems.

Real-World Object-Level Corner Cases

<!--- <img src="https://avs.auto/images/ui-controls.png"> -->

:book: <a href="./paper/"> <b>Corner cases (CC) papers</b> </a>

Long-Tailed Recognition and Corner cases

While the natural data distribution contains head, tail, and open classes, existing classification approaches focus mostly on the head, the tail, often in a closed setting. Traditional deep learning models are good at capturing the big data of head classes; more recently, few-shot learning methods have been developed for the small data of tail classes.

<img width="447" alt="image" src="https://user-images.githubusercontent.com/29084184/179699597-f580c7e5-a98c-46e8-b6ce-ce8ab4e916a4.png">

[1]Ziwei Liu. Large-Scale Long-Tailed Recognition in an Open World. CVPR2019

<br>

🎁 Download

<img src="https://raw.githubusercontent.com/giandbt/synboost/master/display_images/three_anomaly_scenarios.png">

[1] Giancarlo Di Biase. Pixel-wise Anomaly Detection in Complex Driving Scenes. CVPR 2021

<img src="https://coda-dataset.github.io/assets/img/example.png">

[2] CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving

CODA数据集主要是从3个目标检测数据集中提取构建的，包括KITTI、nuScenes、ONCE。从这些公开数据集中选取了1500个场景图片，包含约6000个目标级别的corner case，⬆️ 如上图所示为论文中展示的corner case样例.

Open datasets:

• https://coda-dataset.github.io/download.html
• https://once-for-auto-driving.github.io/download.html#downloads
• https://www.cityscapes-dataset.com/downloads
• https://github.com/waymo-research/waymo-open-dataset

<br>

Corner cases in levels

<img align="right" src="./images/corner_cases_in_levels.png" width="343" title="Do what you like, and do it best!">

Automated driving has become a major topic of interest not only in the active research community but also in mainstream media reports. Visual perception of such intelligent vehicles has experienced large progress in the last decade thanks to advances in deep learning techniques but some challenges still remain.

One such challenge is the detection of corner cases. They are unexpected and unknown situations that occur while driving. Conventional visual perception methods are often not able to detect them because corner cases have not been witnessed during training. Hence, their detection is highly safety-critical, and detection methods can be applied to vast amounts of collected data to select suitable training data. A reliable detection of corner cases will not only further automate the data selection procedure and increase safety in autonomous driving but can thereby also affect the public acceptance of the new technology in a positive manner.

In this work[2], we continue a previous systematization of corner cases on different levels by an extended set of examples for each level. Moreover, we group detection approaches into different categories and link them with the corner case levels. Hence, we give directions to showcase specific corner cases and basic guidelines on how to technically detect them.

Systematization of corner cases on different levels as given in [1]. The theoretical complexity of the detection typically increases from the bottom to the top.

<br> <br>

Some examples

<img src="./images/some-examples-in-levels.png">

Scene 1: A cyclist holds a stop sign in his hand. We don’t know when his going to lift the sign. A driverless car must understand this scenario. Even if he raises the Stop Sign, the autopilot should not stop.

Scene 2: The oncoming vehicles loaded with plastic pipes are scattered all over the place. The autopilot must learn to deal with this sudden situation and avoid their influence on the driverless vehicle.

Scene 3: Due to road construction and other factors, the pavement is covered with cone barrels. The unmanned vehicle must recognize these scenes correctly and realize reasonable driving in the scene full of cone barrels on the road.

Scene 4: The traffic light is green, and the unmanned vehicle has the right of way. Although our unmanned vehicle arrives at the intersection first, it must give way to special vehicles(ambulance, etc) that will arrive later.

Scene 5: When there is a green light at the intersection, the unmanned vehicle is ready to turn left. When encountering the social vehicles running through the red light at high speed, the unmanned vehicle needs to identify this scene and stop in time to avoid the illegal vehicles.

Scene 6: A building facade reflecting cars. How does the machine “understand” that the cars are behind you though you see them in front of you and accordingly instruct the self-driven car to reverse or move forward?

<table class="table table-striped table-bordered table-vcenter"> <tr> <td> <img src="./images/facade-reflecting-cars-1.jpg"> </td> <td> <img src="./images/facade-reflecting-cars-2.jpg"> </td> </tr> </table>

Fig2 A building facade reflecting cars

Scene 7: These scenarios are plenty. Here is one more illustration. How do you read this situation… see figure 2 — is there a building (clock tower) in front of the vehicle or is the vehicle on a road in front of a tram?

<table class="table table-striped table-bordered table-vcenter"> <tr> <td> <img src="./images/tram-example-1.jpg"> </td> <td> <img src="./images/tram-example-2.jpg"> </td> <td> <img src="./images/tram-example-3.jpg"> </td> </tr> </table>

Fig3 Tram Example (Figure 1,2,3)

Scene 8: Traffic Lights

<img src="./images/IMG_5256.JPG">

在自动驾驶中，红绿灯感知模块通常会被当做一个单独的子模块进行单独设计，本文针对红绿灯感知过程中存在的部分问题进行汇总，以达到共同讨论，共同解决行业难点问题的目标。

1. 图像识别问题汇总

【corner case1】故障灯识别（长时间黑灯，长时间黄灯闪烁），远距离容易识别错误问题

【corner case2】闪烁状态识别（红灯闪烁，黄灯闪烁，绿灯闪烁）

【corner case3】远距离条件下，小目标红绿灯识别（高速120km/h下，至少要看到200m远）

【corner case4】太阳光干扰（逆光、光照不足导致图像灯盘颜色过浅，近似黑灯）

【corner case5】抓拍灯干扰（夜间抓拍灯过亮，导致图像过曝，图像灯盘无法分别轮廓边界）

【corner case6】镜头受到污渍干扰（夜间抓拍灯过亮，导致图像过曝，图像灯盘无法分别轮廓边界；雨雪遮挡镜头导致无法抓拍到易识别的图像登）

【corner case7】适配各式各样的交通灯检测识别（竖条灯<常见>，横条灯<常见>，临时竖灯，读条横灯，读条竖灯，高速通道灯，非机动车道灯，人行横道灯，单组灯，两组灯，三组灯，四组灯，箭头灯，圆头灯等）

[1] 读条竖灯

[2] 读条横灯

[3] 三箭头灯

[4] 倒计时与灯盘上

[5] 倒计时在灯盘一侧（通常在灯盘右侧）

【corner case8】交通灯背景干扰（常规背景为天空、树木，不常见的背景包括各式各样的建筑物、广告牌背景）

【corner case9】LED闪烁问题（一些不合规的交通灯存在LED闪烁，人眼看到灯有颜色，图像抓拍到的为黑色）

【corner case10】动态交通灯问题（部分城市的部分交通灯，会根据交通流或者施工情况调整多组灯中灯亮的状态，例如两组灯，一个左转，一个直行灯，在某些时段，只让一个灯亮，容易造成规控无法对应交通灯进行决策，导致该停车不停，该前进不走等问题）

【corner case11】倒计时识别问题（倒计时数字牌往往更小，而且布局各不相同，通常常见的布局包括，单独倒计时牌<立于竖灯右侧或头顶>，与灯牌一体倒计时<立于灯盘中间>；倒计时数字也不相同，通常有个位数字和两位数字，两位数字存在01、02...，最大数字可以到99；夜晚识别倒计时牌容易受到大光晕干扰，造成肉眼无法识别数字；在较远地方，倒计时牌无法识别等问题）

【corner case12】箭头指向、圆头识别问题（较远的地方，箭头与圆头类型容易混淆，箭头指向识别准确度较低等问题）

2. 依赖高精地图识别红绿灯问题汇总

【corner case1】多颗摄像头切换选择问题。自动驾驶中通常至少需要两颗摄像头(远/近焦距)来完成红绿灯图像采集问题，但依赖高精地图方案中，最终只会选择一张图像完成检测识别过程，精确的选择哪一张图像来完成后续过程需要在大量的试用过程中不断调优，从而适配各式各样的场景。

【corner case2】左转待转区采用右前侧宽视角摄像头图像，当左转待转区距离直行交通灯较远时，存在右前侧摄像头成像无法区分红绿灯颜色（即红绿灯在图像上无论是红、黄、绿哪种颜色，肉眼看都是黑色）

【corner case3】左转待转区采用右前侧宽视角摄像头图像，当左转待转区距离直行交通灯较远时，倒计时功能无法正常使用，人眼无法从图像中区分数字

【corner case4】高精地图中是按照车道线lane id与交通灯signal id一一对应绑定，由于检测模型会搜索到感兴趣区域中的全部近似红绿灯灯框，进而再与高精地图通过位置相近计算进行匹配。因此在直行道路，左转待转道路等需要绑定红绿灯的场景中，地图中需要将全部交通灯进行绑定。由于高精地图中绑灯是人工完成的，往往容易出现制图中针对废灯(不亮，一直为黑灯)容易出现忘记绑定问题，最终导致匹配时，可能出现匹配错误问题

【corner case5】高精地图中存在单lane id绑定多signal id问题，规则的不一致性容易导致匹配问题

【corner case6】高精地图中由于最终绑定signal id的lane id往往比较短，但是现实情况要求自动驾驶车辆需要在非常远处能够看到交通灯，因此需要通过对地图中lane id后继车道线进行查询，并且确认后继车道线是