What question did this study set out to answer?

This study aims to improve classification tasks using multimodal neural architectures for heterogeneous and unbalanced data.

March 30, 2026Open Access

Development and Research of Multimodal Neural Architectures for Heterogeneous Unbalanced Data in Classification Tasks

Puntos clave

This study aims to improve classification tasks using multimodal neural architectures for heterogeneous and unbalanced data.
Implemented early and late fusion approaches with cross-modal attention.
Utilized contrastive learning techniques to ensure semantic consistency.
Developed a model that processes features from different modalities using specialized encoders.
Employed adaptive weighting to minimize data imbalance impact.
Applied pruning, quantization, and knowledge distillation to reduce computational costs.
Achieved high accuracy (0.953) and F1-score (0.956) on the BDD100K dataset.
Achieved accuracy (0.956) and F1-score (0.969) on the CMU-MOSEI dataset.
Demonstrated superior classification performance compared to classical methods and SOTA solutions.
Showed consistent results for all classes, including rare classes.

Resumen

The article presents a comprehensive study of modern multimodal neural architectures for integrating heterogeneous and partially unbalanced data in classification tasks. It considers early and late fusion approaches, hybrid architectures with cross-modal attention, and transformers that allow the formation of consistent latent spaces of visual, auditory, and textual features. Particular attention is paid to contrastive learning (CLIP-like approaches, multimodal InfoNCE), which ensures semantic consistency of representations and improves classification accuracy in the presence of uneven data distribution and rare classes. A model is proposed that combines early and late fusion with cross-modal attention and contrastive learning to form a coherent joint latent space. Features of each modality are processed by specialized encoders, and fusion is performed with adaptive weighting, which minimizes the impact of heterogeneous data imbalance and enables the efficient processing of signals of different natures and intensities. The use of pruning, quantization, and knowledge distillation has reduced computational costs without losing accuracy, ensuring stable model performance in real-world streaming scenarios with limited resources. The results of applying the proposed model to the BDD100K and CMU-MOSEI datasets confirmed the model's high efficiency in processing heterogeneous and unbalanced data. For BDD100K, Accuracy 0.953, F1-score 0.956, ROC-AUC 0.947 were achieved, and the integral indicators Micro F1, Macro F1, and Weighted F1 were 0.953, 0.949, and 0.955, respectively; For CMU-MOSEI, Accuracy 0.956, F1-score 0.969, ROC-AUC 0.968, and the integral indicators Micro F1, Macro F1, and Weighted F1 were 0.956, 0.962, and 0.968, respectively. A comparative analysis of metrics with classical methods,SOTA solutions and AutoML (B-T4SA proved that the developed model provides consistently higher accuracy and consistency of classification for all classes, including rare ones, confirming its ability to effectively adapt to high variability and imbalance of heterogeneous data in real conditions.

Me gusta

Guardar

Ver artículo completo