Improving image-retrieval performance of foundation models in gastrointestinal endoscopic images

Introduction: The quality of gastrointestinal endoscopy is verified by documenting specific required images, but identifying these images from the numerous photographs captured during a procedure is tedious. Conventional deep-learning approaches that aim to automate this process are often limited by subjective assessments and poor interpretability. Methods: We introduce a novel content-based image-retrieval framework that employs a dual-backbone architecture, integrating a general-purpose vision foundation model (DINOv2) and a domain-specific endoscopic model (GastroNet). The system is trained using parameter-efficient metric learning to generate discriminative embeddings for efficient similarity searches. The framework is evaluated on 3,500 public endoscopic images (from the Kvasir and HyperKvasir datasets) and validated on entirely unseen real-world and synthetic data. Results: Our model achieves state-of-the-art performance (97.71% Recall@1, 99.14% Recall@5, and 96.74% mean average precision), which is significantly superior to those of single-backbone baseline models. Ablation studies confirm that this improvement is primarily due to the two backbones capturing complementary features. Discussion: These findings demonstrate that the proposed dual-backbone framework offers an accurate and automated tool for assessing the procedural quality of gastrointestinal endoscopy and may facilitate more reliable quality control in clinical practice.