How Evaluation Protocol Shapes Prompt Injection Detector Results: A Study of LLMail-Inject Challenge Submissions

A high score on a randomly held-out set of prompt-injection attempts can reflect familiarity with repeated text, known attackers, or a particular source of benign email. Using Phase 1 of the LLMail-Inject challenge, this study tests how those factors affect reported detector performance. A raw-submission audit measures normalized exact train-test overlap. Separately, a pool of 15,128 unique submission texts is used to compare five evaluation protocols with word and character TF-IDF classifiers and DistilBERT. At an 80/20 split, normalized exact overlap in the raw audit averaged 62.80% across ten random seeds, compared with 4.26% under a chronological split. In the deduplicated model comparison, within-team chronological recall was close to random-split recall, whereas attack recall in the future unseen-team condition was 5.15-6.24 percentage points lower than in the random known-team condition. A paired current-runtime DistilBERT sensitivity experiment increased the input limit from 256 to 512 tokens: future unseen-team recall rose by 0.96 percentage points, but the random-to-future gap remained 5.80 points. Across 55 eligible held-out teams, mean recall was approximately 96%, with a difficult lower tail. On a separate held-out benign-email set, Enron-only training produced a 70.44% false-positive rate, compared with 11.82% for mixed-domain training. Because the future-team comparison also changes test composition and training size, it does not isolate the effect of attacker novelty or time. Detector evaluations should report overlap, group-level recall, and benign-domain false positives alongside aggregate attack recall.