Real-time robotic teleoperation for pavement pothole segmentation, quantification, and localization using multimodal sensing and efficient multi-scale attention-enhanced edge deep learning

This paper proposes a robotic teleoperation pipeline to automate the segmentation, quantification, localization, and visualization of pavement potholes in real-time. The pipeline includes a new attention-based deep learning (DL) model and integrates a 4WD robot, teleoperation workstation, multimodal RGBD sensing fusion and point cloud processing on the edge, and interactive web application through cloud services. The DL model was developed by incorporating an efficient multi-scale attention (EMA) mechanism and transfer learning, which was trained and tested on a pavement dataset with 9472 images. The pipeline was validated through real-world field tests. The new EMA-based DL model yielded a 0.611 mAP50–95(B) and a 0.613 mAP50–95(M), outperforming the YOLOv9 baseline by 8.33% and 6.98%, respectively. The findings also showed that the proposed pipeline successfully automates pothole inspection and generates an interactive map, enabling remote access to the robot's trajectory and detailed pothole information, including pothole area, volume, average and maximum depth. • Robotic teleoperation–enabled pipeline for pothole inspection. • Integration of robotics, RGB-D sensing fusion, and deep learning. • EMA-based deep learning architecture outperforming YOLOv9. • Real-time pothole segmentation, quantification, and localization.