Unexpected object detection based on class correlation in semantic segmentation for automatic driving scenes

The detection of out-of-distribution (OoD) samples in semantic segmentation is crucial for autonomous driving, as deep learning models are typically trained under the assumption of a closed environment, whereas the real world presents an open and diverse set of scenarios. Existing methods employ uncertainty estimation, image reconstruction, and other techniques for OoD sample detection. We have observed that different classes may exhibit connections and associations in varying contexts. For example, objects encountered by autonomous vehicles differ in rural road scenes compared to urban environments, and the likelihood of encountering novel objects varies. This aspect is missing in current anomaly detection methods and is vital for OoD sample detection. Existing approaches solely consider the relative significance of each prediction class, overlooking the inter-object correlation. Although prediction scores (e.g., max logits) obtained from the segmentation network are applicable for OoD sample detection, the same problem persists, particularly for OoD objects. To address this issue, we propose the utilization of the Mahalanobis distance of max logits to evaluate the final predicted score. By calculating the Mahalanobis distance, the paper aims to uncover correlations between different classes, thus enhancing the effectiveness of OoD detection. To this end, we also extend the state-of-the-art segmentation model, DeepLabV3+, to enable OoD sample detection in this paper. Specifically, this paper proposes a novel backbone network, SOD-ResNet101, for extracting contextual and multi-scale semantic information, leveraging the class correlation feature of the Mahalanobis distance to enhance the detection performance of out-of-distribution objects. Notably, our approach eliminates the need for external datasets or separate network training, making it highly applicable to existing pretraining segmentation models.