Long-Term Multi-Resource Fairness for Pay-as-you Use Computing Systems.

Many current computing systems such as clouds and supercomputers charge users for their resource usages. A user’s demand is often changing over time, indicating that it is difficult to keep the high resource utilization all the time for cost efficiency. Resource sharing is a classical and effective approach for high resource utilization. In view of the heterogeneous resource demands of users’ workloads, multi-resource allocation fairness is a must for resource sharing in such pay-as-you-use computing systems. However, we find that, existing multi-resource fair policies such as Dominant Resource Fairness (DRF), implemented in currently popular resource management systems such as Apache YARN <xref ref-type="bibr" rid="ref4">[4]</xref> and Mesos <xref ref-type="bibr" rid="ref23">[23]</xref> , are not suitable for the pay-as-you-use computing systems. We show that this is because of their memoryless characteristic that can cause the following problems in the pay-as-you-use computing systems: 1). users can get resource benefits by cheating; 2). users might not be able to get the total amount of resources that they are entitled to in terms of their resource contributions. In this paper, we propose a new policy called H-MRF, which generalizes DRF and Asset Fairness with the long-term notion. We show that it can address these problems and is suitable for pay-as-you-use computing systems. We have implemented it into YARN by developing a prototype called MRYARN. Finally, we evaluate H-MRF using both testbed and simulated experiments. The experimental results show that there are about $1.1\sim 1.5$ <alternatives><inline-graphic xlink:href="tang-ieq1-2788880.gif"/></alternatives> sharing benefit degrees and $1.2\times \sim 1.8\times$<alternatives> <inline-graphic xlink:href="tang-ieq2-2788880.gif"/></alternatives> performance improvement for users with H-MRF, better than existing fair schedulers. [ABSTRACT FROM AUTHOR]