Modality-task dependent heterogeneous mixture-of-experts model for multimodal time series representation learning

Multimodal time series representation learning has been extensively applied in healthcare, financial forecasting, and autonomous driving. However, heterogeneity among modalities has challenges for robust representation learning and multimodal task adaptation. To address the limitations of existing methods that lack sufficient consideration of modality differences and modality-task dependent interactions, we propose a multimodal learning framework named the Modality-Task Dependent Heterogeneous Mixture-of-Experts (MTD-HMoE) for multimodal time series representation learning. The proposed framework designs an Adaptive Heterogeneous Mixture-of-Experts model (AHMoE). AHMoE employs modality-specific expert sub-networks for time-series, image, and text data to enable semantic feature extraction and learns independent representations within each modality. A pretraining and joint optimization collaborative strategy explicitly models modality heterogeneity by coordinating the optimization of modality-specific output tokens and multimodal self-supervised objectives. In the fine-tuning stage, modality-task dependencies are leveraged to adaptively regulate inter-modal interaction weights, enabling efficient adaptation to diverse downstream tasks under limited labeled data. Extensive experiments on multiple real-world tasks demonstrate that MTD-HMoE consistently outperforms existing approaches, validating its effectiveness in capturing modality heterogeneity and improving task adaptability and scalability.