Heterogeneous Workflow Scheduling Algorithm

As cloud-native applications is more and more popular, job scheduling becomes an important issue. In a cloud-native environment, different jobs have different resource requirements and execution times, causing the efficiency of job scheduling to affect system performance. Therefore, job scheduling algorithms have to face several issues: 1. resource contention resulting from insufficient resource allocation to task groups in decentralized approaches, such as Apache Spark. 2. heterogeneous resource requirements of different jobs running in one computing node. 3. Communication overhead arising from parallel job execution. Therefore, this work proposes a workflow scheduling approach in cloud-native environments. The proposed approach is named as the Heterogeneous Workflow Scheduling Algorithm (HWS), which is a novel solution designed to tackle the complexities associated with task scheduling in cloud-native big data environments. HWS addresses the above issues to avoid the resource contention in group dependencies, and reduce the job execution time and the communication overhead. The proposed approach consists of two primary components: the job priority evaluation module and the job assignment module. Experimental results show that HWS outperforms two heuristic-based workflow scheduling algorithms, MPEFT and IPPTS, with improvements of 11.4% and 11.3% over MPEFT and IPPTS, respectively.