Magnetic resonance imaging (MRI) -guided near infrared spectral tomography (NIRST) is a non-invasive and promising multimodal technique for early breast cancer detection and precise diagnosis. However, existing image-guided NIRST methods have not fully exploited the complementary information provided by MRI and NIRST, which limits reconstruction quality and diagnostic performance. To address this challenge, we propose a hybrid convolutional neural network-Transformer architecture (CNN-Trans) for MRIguided NIRST reconstruction. Specifically, CNN-Trans employs convolutional neural networks (CNNs) to extract local features from NIRST measurements, while Transformers are used to capture global contextual information from MRI images. Dynamic feature fusion (DFF) modules are further incorporated to effectively integrate multimodal features, thereby enhancing NIRST reconstruction quality. Numerical simulation results demonstrate that, compared with three state-of-the-art methods, CNN-Trans reduces the average mean squared error (MSE) by at least 3.81%, while increasing the average peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and total hemoglobin (HbT) contrast by at least 1.81%, 7.23%, and 2.92%, respectively. Furthermore, the CNNTrans trained solely on simulated datasets was directly applied to clinical data from 16 patients without retraining or fine-tuning, and achieving a sensitivity of 90.9%, an accuracy of 87.5%, and an area under the curve (AUC) of 0.909. These results indicate that CNNTrans holds strong potential for clinical application in breast cancer imaging.
Geng et al. (Fri,) studied this question.