Pilot study identifies structural regularities in AI-assisted audit chains, suggesting new predictive approaches in software auditing.
When AI-generated code is audited iteratively across multiple releases, thesequence of findings exhibits structural regularities that are not visible inany single audit round but emerge from the longitudinal chain. We extracteleven such regularities (R1-R11) from a five-finding pilot chain (F22-F26)drawn from a 43-round audit corpus spanning three repositories and fourprimary language substrates. In a pre-registered retrospective experiment,a predictor conditioned on the structured audit-chain log achieved 10/10hits across five dimensions (failing step, error code, substrate layer,fix-size class, and root cause) on two held-out findings. We report thisresult with three explicit caveats: the predictor shared context with thechain's producing instance, the scoring dimensions are coarse (P4 has onlyfour classes; P5 scores mechanism presence without requiring exclusion ofalternatives), and the ablation control separating chain-derived predictionfrom general LLM priors was designed but not yet executed. We also framethe five-dimension prediction as a Hamming-distance scoring problem on afixed-length codeword, which clarifies why coarse categorical hitsaccumulate easily and motivates stricter dimension-weighted analysis. Thecontribution is not the hit rate but the regularities themselves:failure-mode shift, step monotonicity, error-code locality, cross-languageseam recurrence, and seven others. These are properties of the corpus, notof the predictor, and are independently verifiable from the publisheddataset. We release the structured audit-chain log as a Zenodo dataset andpre-register a cross-instance ablation study as the Phase 2 follow-up.
No takes yet. Share an insight, caveat, or question.
Bilal Syed Arfeen (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: