Quality-driven Inference Orchestration for Multi-modality Machine Learning Systems at the Edge

Multi-modality machine learning (ML) systems are increasingly deployed in edge environments, such as smart building and autonomous robotic applications. These systems enable analysis of complex subjects by performing ML inferences on multiple data sources provided by Internet of Things (IoT)-enabled devices. However, operating such ML systems for multi-tenant applications presents a runtime orchestration challenge to optimize the execution of multiple inference tasks across modalities. Analysis requests from different tenants often rely on different data sources with distinct quality requirements, resulting in a wide range of conflicting optimization objectives for runtime inference orchestration. The diversity of ML models and their performance variations further complicate the orchestration, particularly in scheduling inference tasks and distributing inference workloads on a heterogeneous edge system. This paper addresses these challenges by introducing an adaptive orchestration that manages inference task scheduling for multi-tenant applications in a multi-modality ML system. Our orchestration supports a complex trade-off between quality refinements and inference capability of inference services while scheduling multiple inference tasks under dynamic time constraints. The orchestration employs an efficient mechanism for selecting instances of inference services to distribute inference workloads across a heterogeneous edge cluster. This mechanism allows the orchestration to leverage cross-modal information to refine inference quality following the tenant objectives. Furthermore, our orchestration considers the performance variation caused by resource contentions, runtime failures, and explainability overheads while distributing workloads. Extensive experiments on two real-world edge applications in a common heterogeneous edge testbed demonstrate that our orchestration can improve inference accuracy by up to 10% and reduce the late response ratio by up to 18% compared to existing approaches within our multi-tenant edge scenarios.