Reliability‐Aware Star–Galaxy–Quasar Classification in SDSS DR17 : Feature Ablation, Magnitude Robustness, and Probability Calibration

This study presents a reliability‐aware machine learning evaluation for star–galaxy–quasar classification using 150,000 randomly sampled, spectroscopically labeled SDSS DR17 sources. Seven feature configurations were tested to assess the roles of spectroscopic redshift, optical magnitudes, and color indices across five supervised classifiers. Using repeated stratified five‐fold cross‐validation with three repetitions on the development set, the redshift‐free F4 configuration achieved a mean macro‐F1 of 0.8954 ± 0.0017 with Random Forest, whereas the spectroscopic‐reference F7 configuration achieved 0.9848 ± 0.0004 with LightGBM. After development‐only model selection, the corresponding models achieved macro‐F1 scores of 0.8946 and 0.9851, respectively, on an untouched final test set. For the leading ensemble classifiers, the spectroscopic‐reference F7 configuration provided a small but consistent improvement over F6 across repeated validation folds. Beyond aggregate metrics, the redshift‐free F4 model degraded in faint r‐band regimes, particularly in STAR recall, whereas the spectroscopic‐reference F7 model remained robust across most bins of the spectroscopically selected sample. Isotonic calibration improved Random Forest probability reliability without materially changing macro‐F1. Grouped permutation analysis showed strong dependence on the photometric feature block in the redshift‐free F4 model and dominant conditional dependence on spectroscopic redshift in the F7 spectroscopic‐reference model. These findings support reliability‐aware evaluation beyond aggregate accuracy.