A workflow solution for coastal hydrodynamic and water quality modelling using Delft3D Flexible Mesh

Increasingly, high resolution coastal hydrodynamic and water quality models are executed on distributed, heterogenous compute and data infrastructures. Working on these infrastructures presents challenges associated with (1) accessing the requisite model input data, (2) preprocessing the input data so that the models can ingest them, (3) compiling the model for different computing infrastructures, and (4) running the models at scale. These challenges are bottlenecks for setting up and running modelling systems in an interoperable and reproducible way. Here we present a workflow solution for Delft3D Flexible Mesh that was developed for a federated compute and data infrastructure to overcome some of these challenges. The workflow solution is based on snakemake which is a Python, rule-based workflow management system that facilitates creating reproducible and scalable analyses. Snakemake integrates with the package manager Conda and the container engine Singularity and docker such that defining the software stack becomes part of the workflow itself, supporting various operating systems for reproducibility and portability. We demonstrate how it can be used to download the requisite model input data and prepare them for running Delft3D Flexible Mesh on heterogenous computing infrastructures. The workflow ensures reproducibility by clearly documenting the dependencies between the workflow components. Snakemake is able to automatically parallelise the workflow execution by efficiently using available resources and easily adapts from single-core workstations to large clusters, supporting batch systems, and cloud platforms. In the future our approach will facilitate improved automation for coastal hydrodynamic and water quality modelling and forecasting supporting the globally applicable, locally relevant paradigm necessary for implementing a digital twin of the Earth.