When a validation corpus cannot decide: Design effect, calibration invariance and anchor leakage in defence simulation model comparison

Validation studies comparing two computational models over a fleet of physical assets usually treat every record as one observation, and rarely state what prior calibration leaves comparable. We ask what such a corpus can decide. In an interior-ballistics case study (266 muzzle velocities from 19 chambering groups, two lumped-parameter engines), we characterise the calibration operator algebraically, remove anchors at the group level and measure the cluster design effect with the exact record-weighted multiplier. A symmetric family-level correction is exactly invariant to a positive rescaling constant within a family: degrading one model by 11 percentage points leaves its corrected output unchanged to round-off. Anchor leakage, bounded by enumerating 159,936 admissible configurations, was measured once they were reconstructed: it had reversed the historical estimate’s sign. The intraclass correlation is 0.54, and the effective sample size, 21.9. All 12 nominal intervals contain parity, as does the mean absolute percentage error comparison after simulation-calibrated widening; no procedure examined reaches nominal coverage. Under the observed design model, detecting a 1-percentage-point difference needs about 380 chambering groups; 19 cap effective sample size at about 35 regardless of records per group. Validation campaigns should be budgeted in groups, not records.