Modeling and Controlling Deployment Reliability under Temporal Distribution Shift

Machine learning systems deployed in non-stationary environments experience temporal distribution shift that alters the statistical relationship between inputs and outcomes. Existing mitigation strategies such as retraining, recalibration, and drift detection typically optimize average predictive performance at isolated time points, overlooking the temporal stability of reliability during deployment. This paper introduces a deployment-centric framework in which predictive reliability is modeled as a dynamic state consisting of discrimination and calibration components. The evolution of this reliability state over time induces a trajectory whose volatility can be explicitly quantified. We formulate deployment adaptation as a multi-objective control problem that minimizes reliability volatility subject to operational intervention costs. Within this framework, we introduce drift-triggered reliability control policies and characterize the empirical cost–volatility Pareto frontier induced by deployment strategies. Experiments on a large temporally indexed credit-risk dataset demonstrate that selective state-dependent interventions can achieve smoother reliability trajectories while substantially reducing retraining costs compared to continuous retraining strategies. These results show that deployment reliability under temporal distribution shift can be treated as a controllable multi-objective system and provide a principled basis for designing cost-aware intervention policies for real-world machine learning deployments.