ESM-Tools - A modular infrastructure for Earth System Modelling

In the last decades, the operation, maintenance and administration of Earth System Models (ESMs) have become substantially more complex due to the increasing number of available models, coupling approaches and versions, and the need of tuning for different scales and configurations. Another factor contributing to the complexity of operation is the requirement to run the models on different High Performance Computing (HPC) platforms. In this context, configuration tools, workflow managers and ESM-oriented scripting tools have become essential for administrating, distributing and operating ESMs, across research groups, institutions and members of international projects, while still ensuring simulation reproducibility. ESM-Tools is an open-source software infrastructure and configuration tool that tackles these challenges associated with the operation of ESMs. ESM-Tools enables seamlessly building and running ESMs across different HPCs in a reproducible manner. Most importantly, it is used by model developers to distribute standard simulation configurations, so that the user can effortlessly run these predefined simulations while retaining the flexibility to modify only the parameters that align with their specific needs. This lowers the technical threshold for new model users and makes the ESMs more accessible. The source-code consists of an HPC- and model-agnostic Python back-end, and a set of model- and HPC-specific configuration YAML files. In this way, adding a new model, coupled model or HPC is just a matter of writing new configuration YAML files. The configuration files are highly modularized which allows for their reutilization in new setups (e.g. new components are added, while some existing component configurations are reused). Configuration conflicts between the different files are resolved hierarchically accordingly to their configuration category, giving priority to model- and simulation-specific configurations. ESM-Tools also provides basic workflow-management capabilities which allow for plugging in preprocessing and postprocessing tasks and running offline coupled models. The tasks of the ESM-Tools workflow can be reorganized, new tasks can be included, and single tasks can be executed independently, allowing for its integration in more advance workflow manager software if required. Among other coupled Earth System Models, ESM-Tools is currently used to manage and distribute the OpenIFS-based Climate Models AWI-CM3 (FESOM2 + OpenIFS, developed at AWI) and FOCI-OpenIFS (NEMO4 + OpenIFS43r3, developed at GEOMAR, running ORCA05 and ORCA12 in coupled mode with OASIS3-MCT5.0), as well as the AWI-ESM family of models (ECHAM6 + FESOM2). HPCs supported include those of the DKRZ (Hamburg, Germany), Jlich Supercomputing Center (Jlich, Germany), HLRN (Berlin and Gttingen, Germany), and the IBS Center for Climate Physics (Busan, South Korea), with plans to support LUMI (Kajaani, Finland) and desktop distributions (for educational purposes). In this contribution we will introduce ESM-Tools and the design choices behind ESM-Tools architecture. Additionally, we will discuss the advantages of such a modular system, and address the challenges associated with its usability and maintainability resulting from these design choices and our mitigation strategies.