Analyzes prompt injection detection performance under various conditions, revealing key limitations.
We evaluate a two-tier prompt injection guardrail, an Aho-Corasick blocklist followed by a fine-tuned DeBERTa-v3 classifier, and report a sequence of measurements that each revise its apparent performance downward. On the 271-sample corpus its rules were tuned against, the system reaches 1.000 recall at zero false positives. Held out from that same corpus it reaches 0.789. On Open-Prompt-Injection it reaches 0.40, and on BIPIA 0.48. The shortfall is not a tuning deficit. Attacks that announce themselves with override vocabulary are caught almost perfectly, and attacks that inject a well-formed instruction carrying no override phrasing are largely invisible. We then show that one bit of deployment context, whether text arrived as a user turn or as data the application is processing, moves recall on Open-Prompt-Injection from 0.40 to 1.00 at a 0.6% false positive rate, because the same sentence is a legitimate request on one side of that boundary and an attack on the other. Two further results have practical consequences for anyone deploying such a system. INT8 dynamic quantization silently reduced the classifier tier to a constant SAFE prediction, which a fail-open policy rendered invisible. And on NotInject the deterministic tier blocks 0 of 339 benign trigger-word prompts while the learned tier blocks 137, which makes the learned tier unshippable as a blocking stage. Artifacts and every measurement script are public.
No takes yet. Share an insight, caveat, or question.
Prateek Pulastya (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: