Modality-Aware Diffusion Distillation Network for Sentiment Analysis in Missing Modalities

This is the authors' abstract. We don't add key points for this paper.

In this paper, we propose a Modality-aware Diffusion Distillation Network (MDDN), consisting of Diffusion-based Modality Imputation (DMI) and Margin-aware Distillation (MAD) modules, for multimodal sentiment analysis under uncertain missing modalities. More specifically, the DMI module incrementally adds Gaussian noise into the modality-specific distribution space of the available data and recovers missing modalities while adhering to their original distributions, enabling effective imputation of missing data for the network. Furthermore, the MAD module introduces the classification uncertainty of generated samples and available data to reweigh their contributions to the loss function. Consequently, the missing modalities can be reconstructed by MDDN, and be fused dynamically with the existing ones for sentiment analysis. Extensive experiments on the MOSI and MOSEI datasets demonstrate the effectiveness and robustness of the proposed network for dealing with missing modalities in sentiment analysis.