Multi-institutional pan-cancer validation of pathology foundation models for whole-slide image retrieval using TCGA

Foundation models are reshaping computational histopathology, but their value for whole-slide image retrieval relative to strong patch-based and supervised aggregation baselines remains unclear. We benchmarked ten retrieval pipelines on 9387 diagnostic slides spanning 17 organs and 60 diagnoses from The Cancer Genome Atlas (TCGA), using patient-level leave-one-patient-out evaluation within each organ (vertical retrieval), isolating fine-grained subtype discrimination from organ identification. Methods included four pre-trained slide foundation models, a supervised attention-based multiple-instance learning (ABMIL) aggregator trained on patch embeddings, and patch-level retrieval. Performance varied more across organs and diagnoses than across model architectures. Although TITAN achieved the strongest overall results, its advantage was modest: ABMIL—a supervised aggregator trained within TCGA and included as an in-distribution reference—and patch-based methods achieved comparable Top-1 and Top-3 performance, with no consistently dominant model. Morphologically distinctive entities approached ceiling-level performance, whereas rare and closely related subtypes remained difficult to retrieve. Performance was driven primarily by patch-level feature representations; within this benchmark, aggregation provided limited additional benefit beyond strong patch representations. Even the best-performing model achieved only approximately 68% ± 21% retrieval accuracy on TCGA, with multiple subtypes below 50% and some at 0% , highlighting limitations of morphology-based representations and the need for substantial progress before reliable clinical deployment.