U-Net-Based Model Design for Semantic Segmentation of Class-Imbalanced Semi-Synthetic Roads

Accurate semantic segmentation of roads and overlaid markings is essential for multi-camera multi-robot visual localization systems, yet lane markings occupy a tiny fraction of the image area, making them difficult to segment reliably. This paper presents a U-Net design study for semantic segmentation of imbalanced segmentation of a dominant class and two similar, minority classes, that occur on top of the dominant class. We analyze the problem of designing a multi-head U-Net for segmenting semi-synthetic Duckietown model road map images into roads, stop-line markings, and lane-line markings. The multi-head design decomposes the task into a binary road segmentation head and a ternary marking segmentation head, connected through a road-aware loss that restricts marking supervision to predicted road regions. Our work assesses the nine loss functions to approach the class imbalance problem in the marking head—including cross-entropy, focal loss, Tversky loss, Lovász-softmax, and a subset of combinations thereof. These configurations are systematically evaluated on a dataset of semi-synthetic map images generated using an evolutionary algorithm described in a previous work of the authors, where road marking classes are a minority. The Tversky–Lovász combination achieves the highest per-class IoU across all segmentation targets, being statistically significantly better than other configurations. The results demonstrate that the Tversky loss combined with a direct IoU surrogate, Lovász-softmax, is particularly effective for small-object segmentation under severe class imbalance.