RoboFireFuseNet: Robust fusion of visible and infrared wildfire imaging for real-time flame and smoke segmentation

Concurrent flame and smoke image region segmentation is a challenging task, particularly when relying on a single imaging modality. Leveraging the combination of visible (RGB) and infrared (IR) modalities in wildfire imaging significantly enhances the accuracy and robustness of fire segmentation. In particular, during dense wildfire smoke incidents, certain image features are only imaged by one modality. Therefore, the two wildfire imaging modalities are inherently complementary. This paper evaluates the effectiveness of RGB and IR image fusion for flame and smoke region segmentation. A novel intermediate image fusion architecture is proposed, built upon a real-time, state-of-the-art DNN semantic segmentation model, augmented with attention mechanisms that promote efficient image modality fusion. Furthermore, a U-Net-like decoder enables accurate spatial reconstruction of the lower-dimensional encoded features. Practical challenges, such as segmentation robustness in the absence of image registration and sensor failures, are also efficiently addressed. Based on our experiments, the proposed DNN segmentation model greatly outperforms existing multimodal DNN architectures in wildfire scenarios in terms of accuracy, while also comparing favorably to stateof-the-art semantic image region segmentation architectures in general urban datasets. Its real-time capabilities and enhanced robustness render it suitable for robotic applications in dynamic, high-stakes segmentation tasks. The code is available at https://gitfront.io/r/dfotiou/eiTd3o9UURjn/RoboFireFuseNetprivate/.