Leveraging Multimodal Semantic Fusion for Gastric Cancer Screening via Hierarchical Attention Mechanism

Gastroscopy is a widely adopted method for locating gastric lesions and performing the early screening and diagnosis of gastric cancer (GC). However, the effectiveness of traditional GC screening methods depends on the medical skills of the gastroscopy specialist. A lack of knowledge and experience may lead to misdiagnosis and mistreatment, especially in small-scale hospitals. Recently, there has been a significant increase in studies on data-driven computer-aided diagnosis techniques. In this article, we propose a novel intelligent decision-making method for GC screening (ID-GCS), a multimodal semantic fusion-based data-driven decision-making system. ID-GCS exploits a hybrid attention mechanism to extract textual semantics from multimodal gastroscopy reports and performs semantic fusion to integrate the semantics of textual gastroscopy reports and images, resulting in improved interpretability of gastroscopy findings. We evaluated ID-GCS using a real gastroscopy report dataset, and experimental results show that compared with state-of-the-art methods, ID-GCS achieves better sensitivity and accuracy in GC screening.