PROBE: a multi-organism benchmark for evaluating general-purpose large language models on Gene Ontology protein function annotation

This is the authors' abstract. We don't add key points for this paper.

One of the most important unresolved issues in computational biology is systematic protein function annotation. Despite decades of curation by the Gene Ontology Consortium and UniProt Swiss-Prot, most sequenced proteins lack experimentally validated functional annotations. The emergence of Large Language Models (LLMs) trained on scientific literature raises the question of whether they can map natural-language descriptions of experimentally characterized protein function to the correct Gene Ontology (GO) terms without task-specific training. Answering this question requires a benchmark with gold-standard experimental annotations and multi-organism coverage, which existing benchmarks lack. We introduce PROBE ( PR otein functi O n B enchmark E valuation): 1,000 UniProt Swiss-Prot proteins from five organisms, experimental-evidence-only annotations. We evaluate 10 open-source LLMs across five prompt formats, zero-shot and inference-only, yielding 150,000 predictions. The best model (Mistral Large 123B) achieves macro-F1 of 9.3%, yet a most-frequent-5 heuristic outperforms four of the ten LLMs. 41.3% of predicted GO terms were unmatched to the benchmark’s annotations; a temporal-holdout check against current UniProt records found only 1.25% newly supported, suggesting most are genuine errors rather than incomplete ground truth. Performance varied by organism— Saccharomyces cerevisiae unexpectedly scored highest (F1 = 8.6%), but with only five organisms this does not track evolutionary distance, and neither literature density nor annotation-pool size reaches significance, so the finding remains exploratory. Candidate-selection prompting (P5) reduces the unmatched-prediction rate by 33.5 percentage points relative to P1 while achieving the highest F1 (8.2%). GO semantic similarity analysis shows LLM predictions exceed a random baseline by 2.1–2.6× across all namespaces, indicating biologically meaningful semantic alignment despite exact-match failures. A preliminary 50-protein comparison of four closed-source models against the best open-source model found F1 of 0.081–0.127 versus 0.099 (0.028-point gap), suggesting but not establishing that current limitations extend beyond open-source models. PROBE shows that general-purpose LLMs can map some natural-language descriptions of protein function to GO terms, but performance is often insufficient and unreliable for autonomous annotation. Unmatched predictions are common, and organism-level performance differences remain an open question rather than explained by training data or evolutionary relationships. The benchmark, evaluation pipeline, and all 150,000 experimental response records are publicly available to ensure reproducibility.