Exascale workflow design. Deliverable D2.1 of the HORIZON-EUROHPC-JU-2021-COE-01 project MaX (101093374)

The present deliverable reports on the design, architecture, implementation, and development plan for exascale workflows, according to Task T2.1 of WP2.Workflows are the top-level layer in the practical use of a computing resource. Lowerlevel layers are composed of system software, compilers, libraries, and individual codes, and are by now reasonably automatised and/or packaged. In contrast, workflows retain an artisan character as they must be designed for specific use cases, handling the coordination of many tasks involving a variety of applications and possibly heterogeneous resources, and organising the flow of data. While the lower-level computing layers are basically standardised, there is a large degree of fragmentation in the community regarding the tools used for workflow design and orchestration.In this document we provide an overview and a plan for the design and implementation of several exascale workflows, inspired by and targeted to specific scientific showcases. We will not aim at this stage at the creation of a general "workflow development kit", but rather at the combination of a set of tools and ideas to make it easier to handle the scientific demands. We identify common patterns for workflow design among our scientific showcases, and frame the design within the long-running MAX theme of modularity, and exploiting common ideas at different levels of the computational hierarchy.Exascale workflow design is influenced markedly by the boundary conditions imposed on the actual use of the HPC machine(s): the degree of congestion (i.e., whether one can assume that resources are available within reasonable limits) and the needed support for fallback measures in case of system failures and shutdowns. A first design within "ideal conditions" can serve to structure the main ideas and techniques. Further refinement can introduce the needed recovery measures, within limits dictated by a balance between robustness, complexity of design, and the cost involved.