Validation Design Governs the Apparent Performance of Machine Learning Calibration for Low-Cost Optical Particle Counters: A PM1 Study

Low-cost optical particle counters (OPCs) are increasingly deployed to resolve spatial variability in indoor particulate matter levels that sparse reference networks cannot capture, but their output requires field calibration. We evaluated three co-located low-cost sensors against the means of reference nephelometers over seven discrete co-location runs of 20–33 h each, conducted between June and September 2022. We used 1-min and hourly aggregations and six calibration models (multiple linear regression, support vector regression, random forest, gradient boosting, XGBoost, and LightGBM). Reference PM1 averaged 8.4–9.2 µg m−3. Factory-calibrated sensor output underestimated reference values by 5.3–6.1 µg m−3, with an RMSE of 6.5–7.6 µg m−3, representing 74–82% of the reference mean. Under a conventional random 80/20 hold-out, tree ensembles reached R2 = 0.86–0.89 at 1-min resolution. When progressively stricter designs (day-blocked, leave-one-run-out, forward chaining and chronological hold-out) were applied to the same data and models, R2 fell sharply from random to day-blocked to run-blocked validation and remained near or below zero under forward chaining and chronological hold-out. Calibration remained worthwhile, reducing prospective RMSE by 22–48%, but the calibrated slope fell to 0.18–0.44, so real variation was substantially compressed. Validation design accounted for 63–91% of the spread in R2, while algorithm choice accounted for 3.6–10.6%. We conclude that reported machine learning calibration performance is governed more by validation design than by algorithm choice.