QAFHE: A QoS-aware framework forheterogeneous and dynamic edge-to-cloud Kubernetes deployments

Abstract The introduction of the edge-to-cloud continuum paradigm introduces new chal-lenges due to the heterogeneity of computing entities in deployments. Thesechallenges primarily affect the ability to maintain appropriate levels of Qualityof Service (QoS) when assigning workloads and requests to nodes, particularlyconcerning response times, as well as application-specific metrics. Moreover, inscenarios where computing elements are inherently dynamic in terms of avail-ability, computing power, or latency, efficiently assigning workloads to the mostappropriate computing element becomes an even more significant challenge. Cur-rent generic orchestrators, like Kubernetes, have shown themselves to be effectivein homogeneous and static environments, where the usual QoS-unaware schedul-ing strategies focus mainly on load balancing, neglecting aspects such as reducinglatency or constraining application-level metrics. In this study, we reveal thatgeneric orchestrators like Kubernetes fall short when QoS-agnostic policies areapplied to heterogeneous and dynamic edge-to-cloud environments. We introduceQafhe, a novel framework that integrates effortlessly into Kubernetes via a ser-vice. This framework is designed with a set of QoS-aware scheduling policies toeffectively address the heterogeneity and dynamicity found in numerous edge-to-cloud setups. Our experiments, specifically involving the deployment of inferenceservers across diverse nodes, show considerable improvements in response timesacross various dynamic scenarios involving devices with varying capabilities, suchas multi-core CPUs and diverse GPU types.