A Multi-Criteria Sample Selection Framework Using Uncertainty, Reliability, Representativeness, and Non-Redundancy for Emergency Department Prediction

Background: Selecting informative training samples is a fundamental yet challenging problem in predictive modeling, particularly in heterogeneous clinical data. Although supervised learning is typically formulated as an optimization problem over model parameters, the composition of the training set substantially influences generalization performance. In this study, we propose a multi-criteria score-based sample selection framework for a machine learning setting. Method: Four sample-level scores were defined to quantify predictive uncertainty, representativeness, non-redundancy, and reliability. These scores were normalized and combined using three integration schemes: additive weighting, reliability-gated weighting, and rank-based aggregation. For each chief complaint category, a baseline model was trained either on the full training set, on score-selected subsets and on random size-matched subsets. Performance was assessed using the area under the receiver operating characteristic curve (AUROC), the area under the precision–recall curve, sensitivity, and specificity, with classification thresholds determined by the Youden index. Results: Across experiments, integrated score-based subset selection outperformed both full-data training and random subsampling in terms of mean AUROC, while often showing lower variability across chief complaints. Conclusions: The results suggest that sample utility in clinical tabular data is intrinsically multi-dimensional and that explicitly modeling this structure can improve predictive discrimination.