When Reported Coverage Is Not Coverage: A Controlled Audit of Conformal Prediction Implementations

A conformal predictor carries a finite-sample coverage guarantee, but the number a pipeline reports comes from the implementation and can depart from it. This note presents an audit protocol for three choices between the guarantee and the reported figure: threshold calibration, the treatment of empty sets, and the update rule of adaptive conformal inference. The protocol combines existing constructions with an exact coverage accounting identity and a cost rule for the empty-set fallback. Applied to digits, CIFAR-10, covertype and eight public implementations, it produces a consistent picture. A fixed probability threshold yields a set with no coverage guarantee, and its reported figure describes accuracy among non-abstentions; a true-class threshold coincides exactly with split conformal. Forcing the argmax into empty sets raises reported coverage by up to 9.7 points on the image benchmarks, with little effect on a tabular benchmark whose empty rate is near zero. The common alpha_t recursion omits the fixed target and reports higher coverage by inflating the sets; three of the eight libraries use the correct fixed-target update.