Answers are grounded in this paper and cite their sources.
A researcher agrees with an AI auditor that a coded category is unsupported and should be removed. The category is still present in the analysed dataset, and no record explains why. When decisions and data are recorded separately, such cases remain hidden, because neither record shows whether one reached the other. This retrospective study linked the two kinds of records in a completed project in which supervised AI systems coded 11,458 online reviews and 187 complaints about the Singapore–Malaysia confinement nanny market. Selected decisions were traced into the analysed data, and all 276 saved batch files were compared to identify changes that no decision explained. Whether a change could be traced depended on how it was made. Corrections logged by record and field were fully traceable. Recoding was not. Three recoded batches produced 109 changes, and only six of these could be linked to a decision about that record. Thirty-one saved files were also modified later, without their earlier contents being kept. Despite these gaps, none of the counts used in the project’s analysis changed, although only rebuilding them afterwards revealed this. A recorded decision is not evidence that it was carried out. The study proposes records that would provide that evidence, for future projects to test.
No takes yet. Share an insight, caveat, or question.
Huan et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: