Designing benchmarks for data analysis workflow systems

Many scientific disciplines create complex workflows to combine computational and analytical processes into holistic tasks. Workflow systems manage, schedule, and execute these workflows. Due to the high diversity of tasks mapped to workflows, researchers and system developers have built benchmarks to evaluate and compare workflow management systems. These benchmarks target various aspects of workflow management systems or their domain applications. In this work, we propose a design for a comprehensive data analysis workflow (DAW) benchmark for workflow deployments in scientific applications. The goal of the benchmark is to compare the end-to-end latency of each workflow system setup. We analyze the related work on DAW benchmarks and present a summary of the state-of-the-art. We identify the benchmark characteristics of a workflow deployment with a focus on DAWs. Based on our findings, we propose a design for a DAW benchmarking framework that distinguishes between workflow, input data, scalability, latency, and user specifications.