A Controlled Experimental Methodology for Evaluating LLM-Based Vulnerability Repair Using Cascaded Program Analysis Context

Large language models (LLMs) have gained popularity in automatic code vulnerability fixing tasks. However, the effectiveness of LLMs in this setting relies heavily on the quality and structure of the input context. Prior work has shown that LLM repair accuracy improves when guided with static analysis output, with one system reporting fixes for over 80% of reported vulnerabilities when code slicing from static analysis is used. More recently, pairing CPG-derived code slices with Static Application Security Testing (SAST) findings has been shown to help LLMs distinguish vulnerable from non-vulnerable code with F1 scores above 99% on the Juliet benchmark. However, it remains unclear whether the same kind of richer context that benefits detection also translates into better fix generation and how different combinations of context affect repair quality for a fixed LLM under a defined experimental setting. This paper presents a controlled experimental methodology to examine the effect of progressively richer program–analysis context on LLM-based vulnerability repair. Using a cascaded evaluation strategy across four context levels (L0–L3), we evaluate the repair outcomes obtained when static analysis, graph-derived data-flow information, and hybrid analysis are provided as context. The four context levels are: (L0) raw source code only, (L1) source code with SAST findings, (L2) source code with CPG-derived data flow paths, and (L3) a combination of SAST findings and CPG-derived data flow paths. Starting from a pool of 100 vulnerable C/C++ programs from the Juliet test suite covering nine CWE categories, cases that the model fails to fix at each level are escalated to the next. The 100 cases are used as a bounded evaluation sample, and the same LLM is used throughout the experiment to maintain a consistent model configuration. The methodology evaluates each context level using mutually exclusive repair outcome metrics: Correct Fix Rate (CFR), Plausible but Incorrect Rate (PIR), and Invalid Fix Rate (IFR), along with the Recovery Rate at each level and the Cumulative Fix Rate across the cascade. A separate validation experiment applies a three-level cascade (L0; L1 adding SAST findings and application context; L2 adding Dynamic Application Security Testing findings and vulnerability-specific repair guidance) to 53 PHP files drawn from 18 vulnerability modules of the Damn Vulnerable Web Application. Within the evaluated experimental setting, the results show that SAST context (L1) gives the largest improvement over the no-context baseline on Juliet, raising the Cumulative Fix Rate from 54% to 90%. When L1, L2, and L3 are evaluated directly on the same pool of 46 cases that L0 failed to fix, L1 achieves the highest observed repair performance, compared with the CPG-derived code slice approach alone (L2, 83% cumulative) and CPG combined with SAST (L3, 86% cumulative), indicating that, for the evaluated model and cases, adding CPG context to the prompt did not improve repair performance relative to SAST alone. On DVWA, independent validation establishes that the cascade repairs 39 of the 53 files (73.6%); the scanner-based automated classification that drove the cascade returned an upper bound of 52 of 53 (98.1%), thirteen of whose labels did not survive manual review, exploit replay, behavioral smoke testing, or a check that the target weakness recorded for each file matched the weakness its module exhibits. Because the three security-level variants within a module target the same weakness, this corresponds to 18 distinct vulnerabilities rather than 53. A preliminary three-case comparison within a single module, the only module for which both context types could be supplied in isolation, repaired two cases with vulnerability-specific repair guidance alone and none with runtime findings alone. The findings should be interpreted within the scope of the evaluated LLM, selected vulnerability cases, and experimental configuration. They provide empirical observations within this setting and are not intended to establish model-independent conclusions or generalize directly to all LLMs, vulnerability datasets, or real-world software projects.