Leveraging what’s missing to clean what matters most: A fraction of missing information framework for adaptive iterative data cleaning in high-velocity big data environments

Web-scraped price data promise timely consumer price index (CPI) construction but introduce data quality challenges that conventional cleaning may exacerbate rather than resolve. This study investigates the statistical consequences of data cleaning on web-scraped commodity prices, revealing a counterintuitive cleaning paradox wherein automated outlier removal increases inferential uncertainty by eliminating observations that anchor imputation models. Using 7,275 stock-keeping units web-scraped from Zimbabwean online retailers over five months (February–June 2024), we compared uncleaned, automated cleaned, and human-validated cleaned datasets through multiple imputation by chained equations (CART, m = 200 and 50 iterations). Automated cleaning increased mean Fraction of Missing Information (FMI) by 90.7% (from 0.274 to 0.523); human validation partially mitigated this effect (76.5% increase). Laspeyres index validation against official statistics confirmed these findings: uncleaned data achieved MSE of 1.07 (April–June), compared to 1,563 for cleaned alternatives, a three-order-of-magnitude gap persisting across sensitivity analyses. Sequential single-month imputation confirmed that multicollinearity contributes to elevated FMI but does not explain the MSE gap, which originates in asymmetric attrition within the Laspeyres ratio. These findings derive from a five-month, single-country, food-price case study; the underlying attrition-and-anchoring mechanism is likely general, but the reported magnitudes should not be assumed to transfer directly to other commodities, countries, or time horizons. We recommend that National Statistical Offices monitor FMI before and after cleaning, adopt minimal intervention strategies for index construction, and deploy human validation at automated flagging thresholds where classification uncertainty is highest.