This preprint characterises a specific failure mode in synthetic-face detection: a detector can report near-perfect accuracy and near-ideal confidence calibration on its training distribution, and still be silently and confidently wrong the moment the generator behind its inputs changes. The work asks whether post-hoc probability calibration, the standard low-cost remedy for miscalibrated deep networks, can repair that failure. It cannot. Four architecturally distinct backbones (ConvNeXt V2-Tiny, EfficientNetV2-S, Swin V2-Tiny, and MaxViT-Tiny) are fine-tuned to separate StyleGAN-generated faces from real Flickr-Faces-HQ photographs, using the 140k Real and Fake Faces dataset with its native balanced splits. The same models are then evaluated, unchanged, on a far out-of-distribution set of diffusion-generated faces drawn from DeepFakeFace (real IMDB-WIKI photographs paired with Stable Diffusion 1.5 text-to-image and inpainting fakes). Six post-hoc calibrators (temperature scaling, Platt scaling, beta calibration, isotonic regression, histogram binning, and Bayesian Binning into Quantiles) are each fitted on in-distribution validation data only and applied to both test sets, across five seeds, following a cache-once compute-many pipeline. The central finding is an inversion. In distribution, all four models are saturated, with accuracy near 0.999, area under the receiver operating characteristic curve of 1.0000, and expected calibration error (ECE) between 0.0001 and 0.0006, so post-hoc correction has almost nothing to repair. Under generator shift the picture reverses: accuracy collapses to chance, recall on the fake class falls below four percent, and ECE rises roughly a thousandfold to about 0.49 while confidence stays near-maximal, the signature of confident error. No calibrator moves the far out-of-distribution ECE off its collapsed floor, and several methods, isotonic regression in particular, inflate log-loss substantially. A secondary result documents that temperature and Platt scaling preserve ranking to floating-point precision, whereas beta calibration and isotonic regression can alter discrimination on the densely packed shifted scores. The governance-relevant conclusion is a single one: post-hoc calibration certifies the training distribution and nothing beyond it. A probability that gates a decision, rather than a ranking that merely orders inputs, must not be assumed to transfer across generators, and the confidence a detector reports carries no warning of the failure. Responsible deployment therefore requires moving the reliability question upstream into training rather than patching it after the fact. Keywords: deepfake detection, probability calibration, distribution shift, generator shift, responsible AI, uncertainty quantification, expected calibration error, synthetic image detection
Hrushikesh Sanap (Sun,) studied this question.