Answers are grounded in this paper and cite their sources.
Serving ML prediction pipelines spanning multiple models and hardware accelerators is a key challenge in production machine learning. Optimally configuring these pipelines to meet tight end-to-end latency goals is complicated by the interaction between model batch size, the choice of hardware accelerator, and variation in the query arrival process.
No takes yet. Share an insight, caveat, or question.
Crankshaw et al. (2020) studied this question.