Influence of multimodal training data for pipe segmentation in RGB and thermal infrared images in critical infrastructure

In order to protect and secure critical infrastructures, it is important to advance technologies for automated detection and mapping of crucial components like the pipes of fire suppression systems. This work explores the instance segmentation on such pipes in multimodal image media. Instance segmentation has been significantly enhanced by machine learning techniques, but faces complexities when the used RGB training set is limited and lacks diversity. In the means of improving the training efficiency, we study the influence of enhancing the training set with thermal infrared images on the model's performance. In our study, we harnessed both RGB and infrared images captured from the same location. We employed Mask R-CNN with transfer learning from the COCO weights and trained multiple neural networks on different training set combinations with RGB images and thermal infrared images. In order to further enlarge each training data set, we implemented different augmentation methods in the training. Subsequently, we conducted fine-tuning and optimization procedures for the Mask R-CNN training and determined the quality of the pipe instance segmentation of the produced models on test sets containing RGB and infrared images. Using the Jaccard score we provide a quantitative measure of pipe segmentation. The results show that the addition of a comparatively low number of infrared images to the training process does not only improve the detection performance on infrared data, but also improves the model's performance on RGB data. A SOTA comparison shows that the detection quality of our models is comparable with the detection results of Ultralytics' YOLOv8 and Meta's SAM, which are both more recently released.