Answers are grounded in this paper and cite their sources.
Protein functional-class classifiers are usually evaluated with random train–test splits, which can overstate performance when closely related (homologous) sequences fall in both sets. We re-evaluated a widely used benchmark derived from the Protein Data Bank, with 40,211 unique protein sequences in ten functional classes. Six models were compared under random splits and under identity-controlled splits, which keep sequences sharing more than a set identity (90% down to 20%) on the same side: three k-mer composition classifiers, a method that copies the label of the most similar training sequence found by alignment, and two classifiers built on embeddings from the protein language model ESM-2. Under random splitting, 86.5% of test sequences had a training sequence more than 30% identical to them, and a nearest-neighbour model that only copies labels scored in the same range as the learned models. With related sequences kept apart at 30% identity, the k-mer classifiers lost about two-thirds of their macro-F 1 . As a secondary observation, alignment-based label transfer beat an ESM-2 classifier when close training sequences were available, and ESM-2 did better when they were removed; at 30% identity its macro-F 1 advantage was small and not statistically significant. The reason is unclear: almost every benchmark sequence has a close match in UniRef50, the database used to pretrain ESM-2, leaving too few test sequences distant from both the training set and the pretraining data to test generalization beyond them. Random-split results on this benchmark should not be taken as evidence of functional generalization, and comparisons involving protein language models must also account for pretraining data.
No takes yet. Share an insight, caveat, or question.
Alam et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: