Knowing When Not to Predict: Selective Classification under Feature Missingness in Tabular Machine Learning

Tabular classifiers are often trained under complete feature availability but deployed in pipelines where fields can disappear because of sensor failures, unavailable services, incomplete records, or upstream schema changes. This study asks whether model confidence remains useful for identifying errors after such feature loss. A compact, fully reproducible benchmark evaluates Logistic Regression, Random Forest, and XGBoost on three public scikit-learn classification datasets using five repeated stratified 60/20/20 train/calibration/test splits. Models are trained only on complete data and evaluated under entry-wise missing completely at random (MCAR) corruption and structured feature outage at 20% and 40% requested missingness, with five corruption realizations per nonzero condition. Across equal-weight dataset-model-seed cells, mean balanced accuracy decreased from 0.964 on clean inputs to 0.811 under 40% MCAR and 0.805 under 40% structured outage. Mean AURC increased from 0.007 to 0.083 and 0.105, respectively, indicating worse confidence ordering. A confidence threshold selected on clean calibration data for approximately 80% coverage achieved 77.9% coverage on clean tests but only 50.9% under 40% MCAR and 47.8% under 40% structured outage. At 40%, all 18 dataset-model-mechanism cells worsened in balanced accuracy, AURC, frozen-threshold coverage, and frozen-threshold selective risk. The results show that deployment-time feature loss can degrade both prediction and self-assessment, and that abstention thresholds validated on complete inputs may not transfer reliably after feature availability changes.