Benchmarking and simulating scientific workflow systems: a review

In the rapidly evolving landscape of scientific computing, the efficacy of workflow execution is paramount for driving groundbreaking discoveries. This chapter provides a critical examination of benchmarking and simulation methodologies for scientific workflows, highlighting the limitations of traditional benchmarks and showcasing recent innovations tailored to the unique demands of workflow management systems. We discuss why conventional benchmarking approaches fall short in accurately evaluating the complex, interdependent nature of scientific workflows, and present emerging benchmarks specifically designed to assess the performance of (1) workflow management systems, (2) the workflows they execute, and (3) the underlying computing infrastructures, clarifying their distinct yet interrelated roles. We then explore how these specialized benchmarks serve as a foundation for developing high-fidelity simulation frameworks. These frameworks are instrumental in creating comprehensive digital twins of workflow management systems, enabling researchers to model and predict performance across various metrics, including energy efficiency and carbon emissions. By bridging the gap between benchmarking and simulation, we illustrate how these digital twins are becoming indispensable tools for optimizing workflow execution in terms of both end-user performance and environmental impact.