Boundary-Aware Lung Infection Segmentation Through Multimodal Fusion of Chest CT Images and Clinical Text

Pneumonia is a serious lung infection, and its early detection remains challenging. Infection regions vary among patients on computed tomography (CT) images and are difficult to distinguish. Moreover, image analysts overlook clinical descriptions when enhancing image representation and fail to provide a comprehensive representation of CT images. In this study, a modified U-Net model called MLEU-Net is used to segment infection areas in chest CT scans based on clinical description-based tuning of the network layers. These text-based tuning layers help segment subpleural, perihilar, and diffuse regions. The MedSeg COVID Dataset 2 was used. The MLEU-Net model consists of (i) an encoder-decoder structure with skip connections; (ii) an Atrous Spatial Pyramid Pooling (ASPP) module used to capture features at multiple scales; (iii) attention mechanisms combined to focus on important regions; and (iv) dropout layers added to improve stability during training. In addition, features from clinical descriptions are combined with image features to support improved interpretation of the results. The proposed framework was trained on one dataset and externally validated on another. The model shows good performance in detecting infection regions, particularly in subpleural (Dice 0.74) and diffuse areas (Dice 0.71). The perihilar region is harder to detect due to very limited annotated data for that class. The results demonstrate good generalization to unseen data. The proposed multimodal MLEU-Net combines image and clinical information within a boundary-aware segmentation framework. External validation demonstrates that the model generalizes well to unseen CT data and supports reliable pneumonia infection analysis. The proposed framework was externally validated on the MosMedData dataset, demonstrating stable generalization to unseen CT images.