Optimising Job Scheduling in Apache Spark: A Simulation Framework for Algorithm Comparison and Performance Prediction

Key Points

• The research aims to enhance job scheduling efficiency in Apache Spark through a comparative simulation framework.
• Developed a simulation framework to analyze different job scheduling algorithms.
• Conducted performance evaluations based on various workload scenarios.
• Utilized Apache Spark as the platform for testing scheduling algorithms.
• Identified the most efficient scheduling algorithm for resource management in Spark.
• Demonstrated performance improvements over traditional scheduling methods with quantified metrics.
• Revealed significant differences in execution times across tested algorithms.