Abstract 3517: A deep learning-based virtual staining system with multimodal-attention for precision medicine

Abstract Purpose: This study introduces a novel multimodal-attention-based virtual mIF staining (MAS) system designed for efficient, reliable virtual mIF staining from label-free fluorescence images. Our goal was to overcome the performance bottleneck and time-cost limitations associated with traditional mIF techniques and thereby enhance their clinical utility. Method and Materials: Our approach involved developing a cutting-edge MAS model. This model is built upon a sophisticated end-to-end generative convolutional neural network (CNN) architecture. It ingeniously leverages autofluorescence and 4', 6-diamidino-2-phenylindole (DAPI) slides as inputs to generate mIF images. To achieve this, we equipped the model with feature extractors enhanced by pretrained masked auto-encoders (MAEs) and a self-attention combination strategy. These components worked harmoniously to extract antigen-label-related features and precisely locate specific cells within the images. Results: In our comprehensive study, we engaged 94 gastric cancer patients, utilizing the MAS system for automated virtual mIF staining of seven biomarkers, namely CD3, CD20, FOXP3, PD1, CD8, CD163, and PDL1, in both cancerous and non-cancerous tissues. Importantly, the MAS-produced virtual mIF stains matched the quality of traditional manual stains. Furthermore, we validated the prognostic accuracy for gastric cancer using these virtual mIF images, demonstrating their ability to provide clinical information that is as reliable and valuable as that obtained from manually stained mIF images. Conclusions: Our study identifies the MAS system as an important advancement in mIF staining, enhancing personalized medicine with its efficiency and quality. This tool holds significant potential to enable personalized medicine efficiently and cost-effectively, streamlining the process of diagnosing diseases, making prognoses, and developing treatment plans. Quantitative Comparison of Ablation Experiments for MAS system in predicting multiple biomarkers Biomarkers Index U-Net (Avg±Std) ReU-Net (Avg±Std) Att-ReU-Net (Avg±Std) MAS (Avg±Std) CD3 PSNR 27. 053±4. 10 27. 419±4. 15 28. 073±4. 00 28. 546±4. 40 CD3 SSIM 0. 733±0. 07 0. 724±0. 07 0. 728±0. 05 0. 742±0. 07 CD20 PSNR 28. 835±4. 66 29. 244±4. 76 28. 672±4. 52 29. 243±4. 99 CD20 SSIM 0. 921±0. 058 0. 921±0. 057 0. 915±0. 051 0. 923±0. 051 FOXP3 PSNR 31. 769±3. 29 31. 838±3. 10 31. 098±3. 20 31. 862±3. 11 FOXP3 SSIM 0. 613±0. 11 0. 618±0. 11 0. 617±0. 112 0. 617±0. 12 PD1 PSNR 27. 908±4. 47 27. 950±4. 43 30. 283±4. 37 31. 717±4. 17 PD1 SSIM 0. 654±0. 13 0. 659±0. 13 0. 694±0. 12 0. 726±0. 11 Citation Format: Zixia Zhou, Yuming Jiang, Ruijiang Li, Lei Xing. A deep learning-based virtual staining system with multimodal-attention for precision medicine abstract. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts) ; 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84 (6Suppl): Abstract nr 3517.