Duplicate leakage and evaluation integrity on a public synthetic sleep-health tabular dataset: A methodological case study

Public tabular datasets are widely reused to benchmark machine learning for health-related screening, but their evaluation validity is rarely audited. Using the widely cited, synthetic Sleep Health and Lifestyle Dataset (374 records, three classes) as a case study, we show that the very high accuracies reported for it reflect evaluation optimism rather than a property of the prediction task: several published studies report accuracies approaching 98 % , whereas our reproduction under conventional cross-validation reaches approximately 91 % , and confining duplicate records to a single fold reduces this further. Our central finding is duplicate-record leakage: only 132 of the 374 rows are distinct, 64.7 % of rows are exact copies of another row, and the conventional cross-validation used in prior work allows identical rows to fall on both sides of the train/test split. When duplicate records are confined to a single fold using group-aware cross-validation, a gradient-boosted model drops from 91.2 % to 72.6 % , and on fully de-duplicated data to 61.7 % : standard cross-validation overestimates performance by 18.6 percentage points relative to duplicate-group-aware evaluation, and performance falls by a further 10.9 percentage points after full de-duplication, reflecting the reduced effective sample size and conflicting-label structure. Under the corrected protocol we benchmark seven classifiers with correlation-corrected (Nadeau–Bengio) confidence intervals, nested cross-validation for hyperparameter selection, and effect-size–oriented significance testing. Simple models (logistic regression, support vector machine, multilayer perceptron) and CatBoost remain robust ( ≈ 87 – 91 % ), whereas the untuned random forest, XGBoost, and LightGBM show substantial variability, with very wide intervals; after nested tuning they partially recover, underscoring that model rankings on this dataset are not reliable. As a secondary, deliberately negative result, a feature-graph attention network we designed for the task (with a learnable sparse graph and a residual gate) does not outperform a plain multilayer perceptron (full model 88.2 % ± 4.2 vs 89.3 % ± 3.4 ; paired Wilcoxon p = 0.03 ), indicating that graph inductive biases do not provide measurable benefit for this small static tabular dataset. We also report a near-deterministic association between the Occupation field and the label (Cramér’s V = 0.75 ; an 85 % no-learning lookup), which we treat as suggestive of a data-construction artefact rather than as proven leakage. We conclude, in line with recent leakage-aware work in medical imaging, that this dataset is unsuitable for strong claims about sleep-disorder screening unless duplicate structure and synthetic-construction artefacts are handled explicitly; the contribution is a reusable, duplicate-aware benchmarking protocol rather than a clinical result.