What Does Really Count? Estimating Relevance of Corner Cases for Semantic Segmentation in Automated Driving

In safety-critical applications such as automated driving, perception errors may create an imminent risk to vulnerable road users (VRU). To mitigate the occurrence of unexpected and potentially dangerous situations, so-called corner cases, perception models are trained on a huge amount of data. However, the models are typically evaluated using task-agnostic metrics, which do not reflect the severity of safety-critical misdetections. Consequently, mis-detections with particular relevance for the safe driving task should entail a more severe penalty during evaluation to pinpoint corner cases in large-scale datasets. In this work, we propose a novel metric IoU <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">w</inf> that exploits relevance on the pixel level of the semantic segmentation output to extend the notion of the intersection over union (IoU) by emphasizing small areas of an image affected by corner cases. We (i) employ IoU <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">w</inf> to measure the effect of pre-defined relevance criteria on the segmentation evaluation, and (ii) use the relevance-adapted IoU <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">w</inf> to refine the identification of corner cases. In our experiments, we investigate vision-based relevance criteria and physical attributes as per-pixel criticality to factor in the imminent risk, showing that IoU <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">w</inf> precisely accentuates the relevance of corner cases.