Multimodal transformer-based fusion for breast ultrasound: Segmentation and BI-RADS classification

improving reproducibility and diagnostic performance in ultrasound BI-RADS classification using multimodal learning. This multicenter retrospective study included 1231 internal and 498 external breast ultrasound examinations covering BI-RADS categories 1–5. Automated lesion segmentation was performed using advanced deep learning models and benchmarked against radiologist annotations. Quantitative radiomic descriptors, transformer-based deep imaging features, structured clinical variables, and selected genomic biomarkers (clinically established mutation and receptor status indicators) were extracted from segmented lesions. Feature robustness and relevance were ensured through reproducibility filtering and multistage selection strategies. Multimodal data were integrated using a transformer-based fusion architecture with dimensionality reduction. Model performance was evaluated on internal and external cohorts using discrimination, calibration, and clinical utility metrics. The ensemble segmentation approach achieved an external Dice coefficient of 89.9%, comparable to radiologist inter-observer agreement. Radiomic-only models achieved an external accuracy of 71.0% (AUC = 0.717), deep ViT-based features improved accuracy to 78.9% (AUC = 0.808), and genomic features alone achieved 84.0% accuracy (AUC = 0.856). For five-class BI-RADS (categories 1–5) classification, the fused multimodal FT-Transformer model achieved the best external performance, with an overall accuracy of 94.7%, sensitivity of 94.2%, and an area under the receiver operating characteristic curve (AUC = 0.962). The proposed multimodal AI framework enables robust, generalizable, and clinically useful BI-RADS classification from breast ultrasound.