Answers are grounded in this paper and cite their sources.
Background Symptom-based machine learning classifiers for diabetes are commonly developed on the UCI Early-Stage Diabetes Risk Prediction dataset, for which accuracies of 97–99% are frequently reported. The dataset contains many repeated predictor profiles, and their effect on performance estimates has not been quantified. We measured how much row-level cross-validation overstates performance and whether it alters model rankings. Methods Ten ensemble classifiers, logistic regression and a 1-nearest-neighbour reference were evaluated on 520 records comprising 251 unique predictor profiles. Each model was assessed with stratified 5-fold cross-validation repeated 10 times under row-level random partitioning and under profile-grouped partitioning, in which identical profiles never occur in both training and test folds. A sensitivity analysis grouped records by symptom pattern, ignoring age. Hyperparameters were fixed a priori. We report classification, discrimination and calibration measures with profile-level cluster bootstrap confidence intervals and compared models with the Nadeau–Bengio corrected resampled t-test with Holm adjustment. Results Under random partitioning, 68.2% of test records had an identical profile in the training folds. Row-level cross-validation reproduced the accuracies reported in the literature (97.2–98.2% for tree-based ensembles) and overstated the Matthews correlation coefficient (MCC) by 0.105 to 0.157 and the area under the ROC curve by 0.023 to 0.037 relative to profile-grouped cross-validation. The stacking ensemble tied for first place under random partitioning but ranked eighth under grouped partitioning. With grouped partitioning, the highest MCC was 0.857 (95% confidence interval 0.786–0.915; extra trees), which differed significantly only from AdaBoost after adjustment. Grouping by symptom pattern lowered performance further for every model; the 1-nearest-neighbour reference, which matched the best ensembles under profile grouping, showed the largest additional loss. Conclusions Exact and near-duplicate predictor profiles are a large, correctable source of optimism on this benchmark. Studies using it should partition by profile at every level, report calibration and uncertainty, and avoid declaring a single best model without formal comparison. The estimates are internal and do not establish diagnostic or screening utility.
No takes yet. Share an insight, caveat, or question.
Marin-Rodriguez et al. (2026) studied this question.