An accuracy and calibration benchmark of scGPT pipelines and matched classical models for cell type annotation

Background. Single-cell large language models (scLLMs) such as scGPT promise transferable representations for cell-type annotation in single-cell RNA sequencing (scRNA-seq), but their statistical calibration under large-scale supervised within-atlas evaluation is largely unexamined. Poorly calibrated confidence scores can mislead downstream biological interpretation, particularly in clinical contexts. Results. Across six public single-cell atlases (1.3 million cells; triple-negative breast cancer (TNBC), Indonesian peripheral blood mononuclear cells (PBMC), brain, multi-tissue tumour microenvironment, human and pig pancreas) we evaluated frozen scGPT probes (a linear probe and stronger non-linear probes, XGBoost and Random Forest), matched classical baselines (Logistic Regression, XGBoost, Random Forest trained on highly variable genes restricted to the genes present in the scGPT vocabulary), and one end-to-end fine-tuned scGPT experiment on TNBC. Accuracy was reported as Macro F1 with bootstrap 95% confidence intervals; calibration as the Expected Calibration Error (ECE). Classical Logistic Regression on the matched gene set reached Macro F1 = 0.94–0.99 and ECE ≤ 0.013 on every atlas, matching or beating every scGPT pipeline. End-to-end fine-tuning of scGPT on TNBC reached Macro F1 = 0.975 / ECE = 0.003, with bootstrap confidence intervals overlapping those of scGPT + XGBoost (0.972) and below the matched-gene Logistic Regression (0.993). Conclusions. Classical machine learning matched or outperformed every frozen scGPT pipeline we tested on accuracy, and the matched-gene Logistic Regression baseline also exceeded the TNBC end-to-end fine-tuning experiment. On calibration, the matchedgene classical baselines were well-calibrated on every atlas, whereas the standard scGPTembedding+LogReg pipeline was substantially miscalibrated. Calibration metrics should be mandatory in scLLM benchmarking, and classical baselines should be evaluated under matched downstream learners and matched gene sets before any claim of scLLM superiority.