Predictive models are widely used to support decision making, yet they are typically built assuming that future data will follow the same distribution as the training data. In practice, data distributions often change in unseen ways, leading to poor model performance and reduced reliability. This methodological commentary highlights the risks of unseen data distribution shifts and shows how they are frequently overlooked in predictive modeling practice. Drawing on transfer learning, domain generalization, and distributionally robust optimization, we organize existing approaches to handling data shifts and illustrate how uncertainty-aware modeling can be implemented in practice. We conclude with actionable recommendations to guide the design, evaluation, and use of predictive models in uncertain data environments. Our work has implications for policy and practice related to trustworthy and responsible artificial intelligence (AI), predictive modeling, and AI risk management.
Duan et al. (Mon,) studied this question.