Genomic heterogeneity inflates the performance of variant pathogenicity predictions

Recent studies have reported unprecedented accuracy predicting pathogenic variants across the genome, including in noncoding regions, using large deep-learning models trained on vast genomic data. To verify and understand the basis of these reports, we systematically evaluated the prediction accuracy of frontier models over single-nucleotide variants, focusing on general models capable of scoring multiple variant types across coding or noncoding regions. Our evaluations reveal that the performance of genome-wide predictions is severely inflated by mixing variant categories with different baseline prevalences of pathogenic variants. We constructed a benchmark stratified by variant type to identify the best-performing models for each genomic context, determine where reliable predictions remain out of reach, and establish rigorous standards to guide future evaluations.