When Leakage Changes the Conclusion: A Methodological Evaluation of Segmentation-Guided Medical Image Classification

Localise a region of interest, mask the image to it, classify what remains: the design is standard in medical imaging, and whether the mask helps is usually assumed. Testing it on burn-severity grading, we obtained opposite answers from the same images and models. Under a file-level split of an augmented collection, masking won for 20 of 21 architectures (+3.06 percentage points; +2.07 on the eight we analyse further); under a source-grouped split with three seeds it gained 0.55 (95% CI −0.37 to +1.47). The naive split had leaked into one arm only: 80.5% of masked test images shared a source photograph with training, and no unmasked ones did. In a controlled experiment that varies only the leak, with its decision rule fixed before the first run, masking changes accuracy by +0.29 (Swin-Tiny) and −0.10 (ResNet-50) points without a leak. Leaking augmented copies of 71% of the test photographs into the masked arm alone makes it +3.51 and +5.80. Leaking the same copies into both arms makes it −3.27 and −2.54 over training seeds (not resolved once test-set sampling is included), because the same leak inflates unmasked images more: equal leak rates did not make the comparison neutral. End to end, the segmentation-guided pipeline is never ahead of a plain classifier, and it does not transfer to clinical photographs. Eight conventional evaluation choices changed, or could have changed, an answer: only 15 of 120 three-seed subsets of a ten-seed replication detect an external difference that the ten resolve over seeds, and that difference is not resolved once test-set sampling is included; selecting the comparison arm on the test set may account for about 40% of the internal margin, an unresolved estimate. We propose no new method. We report a measured case in which the evaluation protocol, not the architecture, determined the conclusion, with a checklist and a script that recomputes the printed numbers.