When Similar Models Disagree: Model Multiplicity and Geographic Robustness in School Meal Utilization Prediction

Competing predictive models can achieve similar aggregate performance while producing different classifications and downstream conclusions for the same individuals. This model disagreement raises an important question about whether conventional performance metrics adequately capture the reliability of model-based analyses. This study investigates model-choice disagreement across multiple analytical layers using voluntary-response survey data from 8,802 respondents who reported receiving meals through Indonesia’s Free Nutritious Meals program. Logistic Regression (LR), Random Forest, and XGBoost were evaluated using a nested school-disjoint validation framework. All three models were compared in terms of discrimination and probability calibration, while the competitive models were further examined through high-risk classification, predictive attribution, model-based scenario analysis, and province-holdout robustness. LR and XGBoost achieved nearly identical PR-AUC values (0.7270 and 0.7277, respectively), with similarly close pooled PR-AUC under province holdout (0.7288 and 0.7285). Despite this aggregate similarity, the models assigned different high-risk classifications to 5.41% of respondents. Their scenario analyses also produced different threshold-crossing rates: 8.87% of eligible respondents under LR (382 of 4,307) and 6.66% under XGBoost. Random Forest exhibited weaker probability calibration in the fitted configuration and was not retained for downstream comparison. Overall, the findings reveal that model agreement at the population level does not necessarily extend to individual classifications and scenario outputs. LR was selected as the reference model through documented judgment considering predictive performance, calibration, robustness, parsimony, and interpretability. These results demonstrate the importance of evaluating model-choice robustness beyond conventional predictive performance metrics, particularly when model outputs inform downstream interpretation or classification.