Cost-Sensitive Evaluation of Machine Learning Models for Manufacturing Equipment Failure Classification

High accuracy can conceal missed failures in imbalanced equipment data. This study compares logistic regression, a decision tree, and a random forest on 10,000 synthetic AI4I 2020 records using an 80:20 stratified split, five-fold training cross-validation, and a relative cost that weights missed failures more heavily than false alarms. Under default decision rules, the random forest achieves the highest test accuracy (0.9805) and ROC-AUC (0.9653), while the decision tree achieves the highest failure recall (0.6765). At false-negative-to-false-positive cost ratios of 5, 10, and 20, the decision tree has the lowest default-rule costs: 131, 241, and 461. Thresholds selected from out-of-fold training probabilities, without test-label tuning, reverse this comparison: the random forest achieves test costs of 95, 162, and 237. The results show why model selection should consider decision thresholds and error costs together. Because the dataset contains simulated contemporaneous failure labels, these findings concern failure-state classification rather than validated advance warning or demonstrated industrial savings.