Attention-Enhanced Hierarchical Transformer for Multimodal Integration of Mammograms and Clinical Data

Breast cancer has been listed as one of the leading causes of death amongst women all over the world, and the current diagnostic techniques, which are founded on the manual examination of mammograms or individual clinical presentations, are often subjective, neither being consistent nor generalizable. The existing computer-aided diagnosis (CAD) systems are also characterized by significant weaknesses related to poor multimodal integration, no interpretability, and vulnerability to class imbalance. In order to address the inadequacy, the present study introduces an advanced multimodal deep learning framework named Hybrid Graph-Generative Transformer (HGGT), designed to integrate high-resolution mammographic images with the clinical, demographic, proteomic, and histological data pertinent to the patient. The HGGT network is a hierarchical Swin Transformer and CNN-based feature extraction, a Graph Attention Network (GAT) (to identify clinical variable interaction), and a contrastive cross-modal generative fusion system (to match the different modalities). The diagnostic head employs a Bayesian uncertainty-aware classifier to ensure more reliability in the prediction of malignancy. It is trained on 5-fold cross-validation, AdamW, and a cosine annealing scheduler, which is set on Python 3.10. It is demonstrated by the performance of the CBIS-DDSM mammography dataset and a corresponding clinical dataset consisting of over 400 patients that HGGT is much superior with 98.2% accuracy, 98.7% precision, 98.5% recall, 99.2% F1-score, and 99.1% AUC-ROC, having a significant advantage over the established models of ResNet50, EfficientNet-B0 and GAN-enhanced CNN classifier. Overall, the HGGT framework is delivering a scalable, interpretable, and highly accurate diagnosis solution that was a huge improvement over the existing unimodal and poorly integrated CAD system in the detection of breast cancer.