Answers are grounded in this paper and cite their sources.
Large Language Models (LLMs) offer opportunities for automated security code review, but vulnerability detection alone cannot establish evidence reliability or remediation effectiveness. This study introduces Structured Evidence-Centred Review Evaluation (SECR-Eval), a controlled evaluation protocol integrating vulnerability detection, source-evidence localisation, CWE attribution, explanation assessment, engineering-validated repair, robustness, and operational efficiency. Four LLM configurations are compared with CodeQL and Semgrep using PrimeVul, DiverseVul, and CVEfixes, covering 10,000 prevalence-preserving functions, 3600 diagnostic functions, 2000 external functions, 360 explanation cases, 200 repair requests, and repeated-execution assessments. Evaluation considers classification, calibration, false-alert burden, evidence quality, repair validity, stability, latency, and cost. LLM-A achieved MCC = 0.562 (95% CI: 0.516–0.606), top-1 localisation = 0.460, and an explanation score of 6.21/8. Among 82 repair cases with complete validation availability, 27 passed all validation gates (32.9%), representing a confirmed validation yield of 13.5% across 200 requests. LLM-B achieved a lower false-alert burden of 1.04 false alerts per true alert and stronger repeated-execution consistency (94.1% agreement; Fleiss’ κ=0.907). These findings reveal complementary detection and assurance trade-offs, supporting evidence-centred evaluation as a basis for human-supervised DevSecOps security review. However, chronological dataset partitioning does not independently establish repository-disjointness, and benchmark findings cannot demonstrate production readiness or reduced operational security incidents. Future validation requires repository-independent testing, reproducible model configurations, complete repair-assurance pipelines, and controlled CI/CD deployment studies.
No takes yet. Share an insight, caveat, or question.
Aljawabrah et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: