Dual-Branch Dynamic Modulation Network for Hyperspectral and LiDAR Data Classification

Deep learning algorithms that can effectively extract features from different modalities have achieved significant performance in multimodal remote sensing (RS) data classification. However, we actually found that the feature representation of one modality is likely to affect other modalities through parameter back-propagation. Even if multimodal models are superior to their uni-modal counterparts, they are likely to be underutilized. To solve the above issue, a dual-branch dynamic modulation network is proposed for hyperspectral (HS) and light detection and ranging (LiDAR) data classification. Firstly, a novel dynamic multimodal gradient optimization (DMGO) strategy is proposed to control the gradient modulation of each feature extraction branch adaptively. Then, a multimodal bi-directional enhancement (MBE) module is developed to integrate features of different modalities, which aims to enhance the complementarity of HS and LiDAR data. Furthermore, a feature distribution consistency (FDC) loss function is designed to quantify similarities between integrated features and dominant features, which can improve the consistency of features across modalities. Experimental evaluations on Houston2013 and Trento datasets demonstrate that our proposed network exceeds several state-of-the-art multimodal classification methods in terms of fusion classification performance.