Cross-Validation and Normalization in EEG Sleep Staging: Impacts on Generalization, Calibration, and Clinical Validity

Accurate and reliable sleep staging from electroencephalography (EEG) is essential for both research and clinical applications. However, evaluation practices differ widely, and subtle methodological choices can strongly influence reported results. In this study, we examined how cross-validation strategies and normalization protocols affect the reliability and generalizability of EEG-based sleep staging models. Two benchmark datasets, SleepEDF and ISRUC, were used to systematically compare common approaches. We found that record-wise evaluation, often used in the literature, leads to overly optimistic results, while subject-wise and leave-one-subject-out (LOSO) evaluations provide more realistic estimates. On SleepEDF and ISRUC, record-wise median Macro-F1 was 0.70 and 0.71, respectively; under subject-wise it was lower by 9 and 7 percentage points. Similarly, normalization strategies matter: although fold-aware normalization performed better in standard tests, subject-aware normalization combined with test-time adaptation produced the most consistent and clinically relevant outcomes, which improves calibration (lower ECE) and supports safer decisions. In particular, it reduced errors and improved both classification accuracy and probability reliability; for example, on ISRUC, subject-aware further improved Macro-F1 by 0.08, reduced ECE by 0.02, and increased kappa by 0.10, compared with fold-aware normalization. We present a protocol-level, model-independent proof that evaluation and normalization decisions can compete with model selection, particularly when datasets change. Better-calibrated predictions and safer clinical decisions are obtained by using subject-wise/LOSO for internal assessment and subject-aware normalization with test-time adaptation for deployment.