SFN-ESVQA: Spatio-Temporal Frame-Aware Network for Endoscopic Surgery Visual Question Answering

Background: While Visual Question Answering (VQA) models have shown significant promise in the medical domain for developing postoperative review in endoscopic surgery, current models designed for endoscopic surgery question answering exhibit critical limitations. By relying solely on static images, they fail to capture dynamic, time-sensitive features of surgical procedures, often leading to contextually incorrect or irrelevant answers. Aim: This study aims to overcome these limitations by developing a model that leverages spatio-temporal information to improve answer accuracy and relevance, thereby enhancing the efficacy of AI-assisted postoperative review and training. Method: We proposed SFN-ESVQA, a frame-aware network that captures rich spatio-temporal features by integrating video, image, and text modalities. To specifically address the prevalent issue of irrelevant answers, we introduced the Irrelevant Answer Rate (IAR) as a new evaluation metric and designed a loss function to penalize the model for generating irrelevant answers. Result: Our SFN-ESVQA achieved new state-of-the-art (SOTA) performance, outperforming current leading models such as PitVQA-Net and SurgicalGPT. We attained 35.28% and 27.83% improvement in balanced accuracy, 25.44% and 29.86% in F1-Score on both datasets PitVQA, EndoVis18-VQA compared with these models. Furthermore, our analysis with the proposed IAR metric revealed a high rate of irrelevant responses in other prior models. In contrast, SFN-ESVQA demonstrated a superior ability to generate relevant answers. Conclusion: Through multi-modal integration, SFN-ESVQA delivers superior performance and significantly reduces the incidence of irrelevant answering. Our method demonstrates its robustness and significant potential for reliable application in postoperative review and training. Our code is available at https://github.com/tj-messi/SFN-ESVQA.