Comparative Analysis of Transformer Models for Prompt Injection Detection in Autonomous Large Language Model Agents

Prompt injection attacks pose a significant security challenge for large language model systems, particularly autonomous large language model-based agents that may interact with external tools, application programming interfaces, and data sources.This study presents a comparative evaluation of five pre-trained transformer models, namely BERT, RoBERTa, DeBERTa, DistilBERT, and ALBERT, fine-tuned for binary prompt injection classification.A balanced dataset of 4,224 samples was compiled from 10 publicly accessible corpora containing benign and injection prompts and divided into training (2,956), validation (634), and test (634) sets.The models were trained in the Kaggle environment using an NVIDIA Tesla T4 GPU, the AdamW optimizer, and a linear warm-up scheduling strategy.On the held-out test set, RoBERTa achieved the highest classification performance, with 97.48% accuracy, 97.52% macro-precision, 97.48% macro-recall, 97.48% macro-F1-score, and 99.64% area under the receiver operating characteristic curve.DistilBERT achieved the highest reported throughput of 819.13 samples/s and the lowest per-sample latency of 1.22 ms.Statistical analysis using the Wilcoxon signed-rank test and SHapley Additive exPlanations-based interpretability analysis were also conducted to examine model differences and influential linguistic features.The findings demonstrate that fine-tuned transformer classifiers can achieve strong performance on the evaluated benchmark dataset while exhibiting different accuracyefficiency characteristics.However, the results should be interpreted within the scope of the evaluated data distribution, as the study does not include independent external datasets or adaptive adversarial attacks.These limitations indicate that further validation is required to establish generalizability and robustness in real-world prompt-injection defense scenarios.