Answers are grounded in this paper and cite their sources.
Autoscalers for large language model (LLM) inference clusters that add capacity when a predicted overload probability crosses a threshold depend on the probability scale, not only on the ranking. This simulation study evaluates five such predictors under sparse telemetry and runtime anomalies on two latent-state generators and a request-level serving simulator with queueing, service-level objectives (SLOs), startup delay, and modeled power. With 3-period instead of 50-period histories, models trained on long histories became overconfident (expected calibration error up to 0.099) or conservative, whereas a Markov filter stayed calibrated and retraining on short histories removed most of the error. Temperature scaling fitted on long histories did not transfer; refitted for short histories, it cut this error by up to 68.7%, removed all 290 wrong high-confidence predictions of one model by lowering their confidence, and cut the overconfident models' idealized unmitigated overload by 59.8–82.8% at the cost of more idle capacity. With a 20-s startup delay in the request-level simulator, learned predictors used 24.0–25.3% less modeled energy than always-on scale-out, with a 1.0–1.2 percentage-point increase in the SLO-violation rate, and 95.3% of scale-out-induced violations occurred on slow new replicas. Calibration should therefore be validated per telemetry regime and checked against request-level outcomes.
No takes yet. Share an insight, caveat, or question.
Mingxuan Tao (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: