Multimodal diagnosis of flatfoot using transformer and graph-based deep learning Models

Flatfoot is a common musculoskeletal disorder that, if not detected in its early stages, can lead to impaired posture, reduced mobility, and diminished quality of life. Conventional diagnosis relies heavily on manual interpretation of radiographic and clinical findings, which can be time-consuming, subjective, and dependent on expert experience. This study aimed to develop a multimodal deep learning framework that integrates complementary anatomical and functional information for automatic flatfoot classification. The proposed framework jointly processed three data modalities: (i) RGB plantar foot images, (ii) plantar-pressure time-series signals collected during gait, and (iii) biomechanical angular measurements. Three modality-specific branches were employed: a Swin Transformer to extract spatial representations of plantar morphology, a bidirectional long short-term memory (BiLSTM) network to model temporal plantar-pressure dynamics, and a Graph Convolutional Network (GCN) to represent structural relationships among biomechanical measurements. The modality-specific representations were projected into a common latent space and integrated using a multi-head attention-based fusion mechanism. The framework was evaluated using participant-level stratified 10-fold cross-validation on a clinically acquired multimodal dataset comprising 978 foot-level samples from 489 independent participants. Complementary publicly available pressure-related data were used exclusively during model development and were not included as independent multimodal cases in the final evaluation. The proposed multimodal framework achieved an accuracy of 94.0%, an F1-score of 0.94, and an area under the receiver operating characteristic curve (AUC) of 0.96. The integration of visual, temporal, and biomechanical representations demonstrated the effectiveness of combining complementary information within a unified multimodal framework for flatfoot classification. The findings demonstrate the potential of transformer- and graph-based multimodal deep learning for computer-aided flatfoot diagnosis. By integrating plantar morphology, temporal pressure dynamics, and structured biomechanical information, the proposed framework achieved high classification performance. Further external validation using independent multi-center datasets is needed to assess its generalizability, and future studies may extend the framework to multi-class foot deformity assessment.