How Much Human Review Does AI Data Entry Need? Guaranteed-Error Automation with Conformal Risk Control

Organizations that use machine learning to automate data entry face a practical question: which predictions can be accepted automatically and which must be checked by a person? A common answer is a fixed confidence cut-off, such as accepting every prediction made with at least 90% confidence. This paper evaluates that practice against threshold-selection procedures with a statistical guarantee on the error rate of automatically accepted records. Using four public datasets (BANKING77, CLINC150, AG News and Adult), eight dataset–model combinations and 500 random calibration/test splits each, we find that a fixed 0.90 cut-off is unreliable: it automated between 18.5% and 82.5% of records depending on the model and, for two combinations, exceeded a 2% error target in every run. Choosing the threshold from a sample of checked records without a safety margin exceeded the error target in 34–57% of runs, and collecting more checked records did not fix this. In contrast, a Learn-then-Test procedure with exact binomial p-values, which we combine with a hybrid multiple-testing scheme, exceeded a 5% error target in at most 3.6% of runs (the allowed rate was 5%) while automating 58–85% of records using 1,000 checked examples. About 500 checked records were needed before useful automation became possible (63% of records automated on average), and 1% error targets could not be certified with 1,000 records. When 20% of live inputs were of an unexpected type, a threshold calibrated on clean data exceeded the 5% target in 95–96% of runs, whereas calibrating on a sample of realistic traffic restored the guarantee. We conclude with practical guidance for deploying human-in-the-loop automation with a verifiable error budget.