The calibration-to-accuracy mapping in P300 spellers does not transport across cohorts and was not restored by tested alignment or local recalibration

Objective A calibration-derived score has been related to online P300-speller accuracy in the same session, always within the cohort measured. We evaluated whether a mapping from that score to expected accuracy transports to withheld cohorts. Approach Retrospective secondary analysis of BigP3BCI 1.0.0, the largest evaluation of this relationship to date: 18 of 20 studies contributed 271 participants, 739 session-condition records, and 19,611 character selections. The predictor was a decoder's cross-validated discriminability on calibration epochs, one study withheld from development. Between-cohort variation was summarised by the random-effects standard deviation tau. Main Results The association was positive in all 18 cohorts but varied widely (r 0.190 to 0.928; participant-level r = 0.714, p < 0.001), with 12 CIs excluding zero. Pooled estimation error was 0.098 (95% CI 0.091 to 0.107) against a benchmark of 0.146. Calibration did not transport: intercept tau 0.87 (unrepresented-cohort interval -1.97 to 1.85, log-odds); cohort-specific slopes varied more than tenfold (tau 0.43, interval 0.111 to 2.005). Neither alignment (signal- or score-space) nor either nonlinear boundary meaningfully reduced this heterogeneity. Local recalibration with up to 16 participants did not reliably improve on transporting; refitting both intercept and slope was significantly worse than transporting at the smallest sizes. Median time per selection, a stopping-rule proxy, accounted for 66% of between-cohort slope variance in meta-regression. The four ALS cohorts gave an uncorrected slope spread of 0.223, against 0.560 overall. Significance The evaluated score is a classifier's cross-validated discriminability on each session's calibration data rather than the source studies' undocumented online decoder. That score reproducibly signals within-session accuracy, but its mapping to accuracy is cohort-specific and supports ranking sessions after local validation rather than reporting accuracy elsewhere. The four-cohort ALS subgroup appeared more homogeneous than the full archive, illustrating how conclusions about transportability from a small cohort set can be fragile. .