Improved classification of agricultural injuries from free-text analysis of pre-hospital care reports under temporal drift

Agriculture, Forestry, and Fishing occupations have the highest fatality rate of any US sector, yet nonfatal injuries remain poorly captured. Machine learning can identify agricultural injury cases from free-text pre-hospital care records (PCRs), but surveillance systems operate for years, and data distribution shifts with time. We benchmark four classifiers for case detection, show how their accuracy declines under this temporal drift, and evaluate ways to manage it. Using a manually coded dataset of PCRs from Maine, 2008–2022 ( n = 60,010), we benchmarked Naïve Bayes, penalized logistic regression (elastic net), a support vector machine (SVM), and gradient-boosted trees (XGBoost) under a temporal design (train 2008–2016, test 2018–2022). Discrimination was summarized by AUC (with 95% confidence intervals) and by the false-positive rate needed to capture 90% of cases (FPR*). Drift was quantified with the population stability index. We evaluated three ways to counter it: re-estimating the decision threshold on the most recent labeled year, retraining on all data to date, and fine-tuning the existing model on recent data (transfer learning). On stable-period data (2008–2016), XGBoost, elastic net, and SVM performed almost identically (test AUC ≈ 0.94), while Naïve Bayes matched them on discrimination but needed a higher false-positive rate to reach 90% capture. Trained on 2008–2016 and tested on each later year, performance held through 2019, then declined across 2020–2022 as feature distributions shifted and the labeled case fraction rose, the latter driven substantially by how records were selected for labeling. A decision threshold set to the year prior to test year could not hold the 90% target, over-capturing review burden in some years and missing cases in others as the score distribution shifted relative to the carried-over cutoff. Retraining and transfer learning gave only modest recovery, indicating genuine drift on sparse recent data. The three leading classifiers were statistically indistinguishable year to year. Temporal drift substantially degrades free-text injury classification and is not fully reversed by retraining on limited recent data. For deployment, pairing a fast-to-retrain classifier with continuous drift monitoring and threshold re-estimation on more recently coded data is more robust than a fixed model and threshold. We propose a practical protocol for detecting and managing drift in surveillance machine learning. Not applicable.