Acceptable GPCR–nanobody predictions can mask errors in confidence ranking

Preprint; not peer reviewed. This 30-page PDF contains the manuscript and supplementary material. The underlying data, analysis code and prediction outputs are archived separately at https://doi.org/10.5281/zenodo.23523297. This manuscript copy updates the data-availability statement to reflect publication of that dataset; the scientific content is unchanged from the manuscript accompanying the dataset. Structure predictors can return many models of one GPCR–nanobody complex, leaving the user to choose one, usually by confidence. We used Boltz-2 to generate batches of 25 models for complexes with known structures, without supplying experimental coordinates. After confidence selected a model, we compared every prediction with the experimental structure using DockQ (0–1, higher is better; ≥0.23 denotes acceptable docking quality). For GPR158, 24 of 25 models were acceptable, yet confidence selected the model ranked 24th by DockQ, 0.130 below the best generated. Acceptable-or-unacceptable reporting would count this as a success. ADGRV1 and MC4R produced no acceptable baseline models. A follow-up chosen after these failures generated 100 additional predictions: broader sampling produced one borderline acceptable ADGRV1 model, ranked 17th by confidence; the other three new batches produced none. Development comparisons used 1,200 predictions to test generation settings and scoring rules learned from confidence and contact measurements. Initial learned rules worsened selection. The largest average gain in the expanded comparisons was 0.017 DockQ, below the 0.020 minimum set in advance. No learned rule qualified for testing on the reserved complexes. Across 48 development batches from 11 receptor groups, confidence selected models averaging 0.375 DockQ, compared with 0.347 expected from random selection, giving each group equal weight. Confidence still left a mean gap of 0.088 to the best model. Among its 30 acceptable choices, the median gap was 0.065. These retrospective results from repeated receptor groups do not establish performance on new receptors. Confidence remained the baseline under our criteria. Benchmarks should report continuous accuracy of both selected and generated models, alongside acceptable-selection counts. Dong Wang Darwin Cai is employed by Hansoh Pharmaceutical Group Company Limited. This work was conducted independently, without company funding, resources, or supervision. This work received no specific funding.