From Model Performance to Audit Evidence: An Experimental Study of Purpose-Specific Validity in Algorithmic Audit Procedures

Audit firms validate technological tools before approving them, and the IAASB's proposed ISA 500 (Revised) leaves whether a tool achieves its intended purpose largely to the firm's quality management. This working paper asks to what extent conventional validation results justify relying on a tool's outputs for a specific assessed risk, and what further evidence is needed when performance may not carry over to the engagement. It distinguishes three conditions: the reliability of the input information, the correctness of the computation, and purpose-specific validity. Experiments on a public synthetic general ledger test three implications. Study 1 finds that, for risks the firm's validation covered, relying on an approved tool for a specific risk was unjustified in 84% of decisions under aggregate validation and 2% when validation was organised by risk, with no justified reliance forgone; for risks it did not cover, an engagement-level known-answer test cut unjustified reliance from 67% to 3%. Study 2 finds that effective tools stayed effective under four planned changes while monitoring signals raised false alarms; in an exploratory scenario, a tool refitted each period lost most of a risk type while its exceptions, fixed in number by the review budget, kept their usual spread across accounts. Study 3 finds that known-answer tests estimate detection within 0.05 on average, erring low for tools that rely on rarity, and falsify unfit tools cheaply, but speak only to cases like those seeded; remainder samples of affordable size rarely reveal a wholly missed condition. The rates describe one synthetic generator and one suite of tools; they show mechanisms, not their prevalence in practice. The paper proposes draft application material for ED-500 and ED-330. Not peer reviewed. The studies were specified in advance in a versioned design document with a dated change log; the document was not deposited with a third party before the studies ran. One scenario and two checks were added after results had been seen, and are labelled exploratory or post hoc throughout. The author draws on this study in responses to the AUASB and IAASB consultations on the proposed ISA 330, ISA 500 and ISA 520. How this was made: the author set the research question, designed the studies and made every analytical and editorial decision, and is responsible for the content. Code, analyses and drafts were prepared with Claude (Anthropic) under the author's direction, and each substantive claim was traced to its source in a claims ledger. Data and attribution: the experiments use the DIB General Ledger Data by Gronewald, Stephan, Rombach and Fettke (2026), Zenodo, doi:10.5281/zenodo.22014653, licensed CC BY 4.0, together with an adapted version of its generator notebook, used under CC BY 4.0. The changes are listed in the code, which is available from the author on request.