Evaluating Supervised Machine Learning Models: Principles, Pitfalls, and Metric Selection

Reliable evaluation of supervised machine learning models requires more than reporting a standard set of performance metrics. Although extensive guidance exists on individual metrics and validation techniques, this knowledge is often considered in isolation, making it difficult to determine which evaluation choices are appropriate under different data characteristics and operational objectives. This review and tutorial addresses this gap by presenting a unified, decision-oriented perspective on model evaluation that connects prediction task, dataset structure, error consequences, validation design, and metric selection. Rather than providing only a catalogue of evaluation measures, the paper combines methodological discussion with controlled empirical scenarios across diverse classification and regression datasets to demonstrate when and why commonly used metrics can lead to different interpretations of model performance. The scenarios examine class imbalance, asymmetric misclassification costs, multiclass averaging, probability calibration, outlier sensitivity, residual behaviour, and the instability of percentage-based errors. Particular attention is given to cases in which apparently strong performance under one criterion conceals important weaknesses under another. Through these examples, the paper provides practical guidance for selecting and interpreting complementary metrics and validation strategies according to the statistical properties of the data and the consequences of prediction errors. The resulting framework positions model evaluation as a context-dependent decision process rather than a post hoc reporting exercise, providing practitioners and researchers with a structured basis for designing more reliable, transparent, and operationally meaningful evaluations of supervised machine learning models.