Length, Not Similarity: What an Expert's Suspicion Tracks in Code-Similarity Screening, and How Much of It Is the Instrument

A code-similarity screen only becomes a decision once a human calls a pair a copy rather than convergence, and that judgement — not the detector — is what every operating threshold is calibrated against. This paper takes the judgement as the object of study. On a full platform-year of auto-graded work (10,849 submissions in seven languages from 172 students) one expert rated a 45-pair stratified sample under a written protocol, setting the 100-token operating gate on which a companion paper's screening rates depend. Four results follow. (i) What separates the rater's classes is solution length, not the similarity value: pooled over the 42 decided pairs the AUC is 0.950 [0.865, 1.000] for length against 0.438 [0.250, 0.669] for the score, and restoring the identifiers, literals and comments the metric discards does not rescue it (0.564). Restricted to the operating band the two are indistinguishable and n = 24 cannot resolve whether either ranks pairs, so what survives is the weaker claim that the value is nowhere demonstrated to rank pairs. (ii) That length effect cannot be separated from the instrument that named it: the protocol told the rater to condition on length and printed each pair's token count, and re-rating with both removed halves the above/below-gate difference in substitute raters (+0.34 to +0.17), while a length-stratified 100-pair holdout shows raters responding to a gradient rather than to a step at 100. The effect is reported as jointly a property of the code and of the instrument, split unidentified. (iii) Three locally run language models did not reproduce the written criterion — agreement with the expert barely distinguishable from chance, every interval admitting zero — while each was highly self-consistent, so the rules differ rather than the decoding being noisy. (iv) Because the sample was drawn only from already-flagged pairs, verification bias is complete: false and true negatives are unidentified, and holding the measured flag rate fixed, positive predictive value runs from 0.09 to 0.82 across assumed prevalences. Stated limitation. One human rated, and it was the author, who also designed the tasks, taught the course and is affiliated with the platform supplying the data. The protocol's own double-rating requirement stands unmet and nothing in this paper repairs it. What the paper contributes is an account of what such a calibration does and does not license. Data. No student source code is contained in or reconstructible from this document; it reports aggregate statistics only. The underlying extract is pseudonymised, not anonymised, and is not released. The analysis pipeline, the expert and model rating labels, both rating prompts with their hashes and the processed pair-level files are archived separately at doi:10.5281/zenodo.20756830.