Multi-Modal Learning with Missing Modality Data via Hybrid Prompt Learning

The success of large-scale image-text multi-modal pre-training has demonstrated the potential of combining multiple modalities. However, other multi-modal domains such as vision-audio and vision-acceleration, constrained by the difficulty of data collection, struggle to benefit from large-scale pre-training. Additionally, existing methods fail to address the issue of modality missing that frequently occurs in real-world scenarios. This paper achieves multi-modal learning for vision-audio and vision-acceleration data by utilizing a large-scale pre-trained vision model. To process 1D signals, we employ the short-time Fourier transform to convert them into 2D spectrograms. To fine-tune the model, this paper proposes the Hybrid Prompt Learning method, which handles both single-modal and multi-modal input through a hybrid prompt token pool and leverages a modality-shared adapter to achieve joint optimization of the model with single-modal and multi-modal data. Experiments on audio-visual event localization and rail surface condition recognition tasks are conducted. Our method outperforms all the compared methods by a clear margin. Our design also enables the inference with both single-modal and multi-modal data. These results illustrate the promising ability of our method.