Benchmarking and Rethinking Evaluation of Pre-trained Language Models for Automated Vulnerability Patching

Software vulnerabilities pose critical security risks, demanding prompt and effective mitigation strategies. While advancements in Automated Program Repair (APR) have primarily targeted general software bugs, the domain of vulnerability patching, which is a security-critical subset of APR, remains underexplored. This paper investigates the potential of pre-trained language models, CodeBERT and CodeT5, for automated vulnerability patching across diverse datasets and five programming languages. Rather than reporting accuracy alone, we examine three factors that determine how such results should be interpreted: how preprocessing-induced information loss affects accuracy, how well commonly used evaluation metrics agree with one another, and how patch complexity relates to model performance. We find that preprocessing loss significantly degrades BLEU scores for both models, while structural and exact-match metrics remain largely unaffected, indicating that preprocessing artifacts disrupt surface-form similarity more than deeper correctness. Evaluation metrics themselves show weak, model-dependent agreement, with no dataset showing consensus across metrics on which model performs better, revealing that metric choice alone can change reported conclusions. Patch complexity is consistently associated with lower accuracy across both models, though this relationship is only statistically robust for CodeBERT, and no significant difference in complexity sensitivity between the two models could be confirmed. Together, these findings benchmark model performance while highlighting key limitations in both the models and the metrics used to evaluate them, offering guidance for more reliable evaluation of automated vulnerability patching in practical security applications.