Integrating UAV-acquired RGB and LiDAR point cloud data using multimodal learning for wheat lodging mapping and area estimation

Wheat lodging, which occurs when wheat stems bend or fall to the ground, significantly impacts both yield and quality. Timely and accurate mapping of lodging areas is crucial for assessing severity, guiding the selection of resistant varieties, and evaluating agricultural losses. While unmanned aerial vehicle (UAV)-acquired RGB imagery, combined with deep learning based networks, has shown promising potential, existing methods typically rely solely on RGB imagery, which may be insufficient for capturing key characteristics of lodged wheat, such as height variations compared to healthy wheat. In this paper, we propose integrating LiDAR-derived Digital Surface Models (DSMs) with RGB imagery, and introduce a novel multi-modal fusion network, namely MFWLNet, for mapping wheat lodging and area estimation. Specifically, MFWLNet utilizes a dual-branch encoder architecture, incorporating both a Transformer and a standard CNN structure, to extract rich multimodal features of wheat lodging. A Local-Global Cooperative Fusion (LGCF) module, featuring channel and spatial attention mechanisms, ensures feature complementarity at multiple scales across the branches, enabling local and global feature interactions. Additionally, a dynamic gate fusion strategy is designed to enhance computational efficiency and promote network lightweight. Furthermore, a cross-modal fusion approach is proposed, consisting of cascaded components: the Cross-Modal Adaptive Rectification Module (CM-ARM) and the Dynamic Feature Fusion (DFM) Module. This approach facilitates enhanced information interaction between features and improves the quality of the fused features for accurate lodging depiction. Extensive experiments on two UAV based wheat lodging datasets demonstrate that MFWLNet outperforms both unimodal and existing multimodal methods, offering an efficient solution for wheat lodging mapping using UAV-based remote sensing. This study provides a new perspective that advances traditional approaches relying solely on RGB imagery, advocating for the integration of multi-source data in crop lodging monitoring.