Automatic segmentation of ultra-high resolution multi-sensory LiDAR and image data: A cross-fusion deep learning algorithm with dynamic channel selection

Inclusive cities need accessible public spaces for all residents, including those with disabilities. Achieving this relies on precise data about urban features like sidewalks, curbs, and streets. While mobile LiDAR and high-resolution cameras now support detailed citywide surveys, managing and analyzing these extensive multi-sensory datasets remains a challenge due to high computational demands. To overcome this, we present Y-Net, an automated semantic segmentation approach designed for high-resolution multi-sensory data. Y-Net uses two parallel encoder networks—one for RGB images and another for projected LiDAR geometry—connected through mid-level cross-fusion and strengthened by a Feature Selection Layer (FSL). The FSL is a parameter-free mechanism that efficiently highlights the most relevant feature maps at each stage by averaging feature intensities over time. Tested on a 200-meter contiguous segment, Y-Net achieves 96.92% overall accuracy and 70.15% mIoU, effectively identifying sidewalks (84.88% IoU), roads (93.58% IoU), and buildings (88.79% IoU). Ablation studies with three different seeds show all components significantly improve results, with the FSL adding 3.32 mIoU points without increasing parameters. Y-Net outperforms a 6-channel U-Net by 5.30 mIoU points. Although Point Transformer v3 has a higher overall mIoU (72.29%), Y-Net surpasses it on key accessibility features like sidewalks (84.88% vs. 63.05%) and doors (27.26% vs. 20.64%), while using a fixed input size of 512×512 and a peak memory of 6.38 GB.