Long-Term Multi-Resource Fairness for Pay-as-you Use Computing Systems

Many current computing systems such as clouds and supercomputers charge users for their resource usages. A user's demand is often changing over time, indicating that it is difficult to keep the high resource utilization all the time for cost efficiency. Resource sharing is a classical and effective approach for high resource utilization. In view of the heterogeneous resource demands of users' workloads, multi-resource allocation fairness is a must for resource sharing in such pay-as-you-use computing systems. However, we find that, existing multi-resource fair policies such as Dominant Resource Fairness (DRF), implemented in currently popular resource management systems such as Apache YARN 4 and Mesos 23, are not suitable for the pay-as-you-use computing systems. We show that this is because of their memoryless characteristic that can cause the following problems in the pay-as-you-use computing systems: 1). users can get resource benefits by cheating; 2). users might not be able to get the total amount of resources that they are entitled to in terms of their resource contributions. In this paper, we propose a new policy called H-MRF, which generalizes DRF and Asset Fairness with the long-term notion. We show that it can address these problems and is suitable for pay-as-you-use computing systems. We have implemented it into YARN by developing a prototype called MRYARN. Finally, we evaluate H-MRF using both testbed and simulated experiments. The experimental results show that there are about 1.1 ~1.5 sharing benefit degrees and 1.2× ~ 1.8× performance improvement for users with H-MRF, better than existing fair schedulers.