Multi-cloud processing with Dask: Demonstrating the capabilities of DestinE Data Lake (DEDL)

The DestinE Data Lake (DEDL) is one of the three key system components of the Destination Earth (DestinE) initiative of the European Commission. It is in the responsibility of the DEDL to provide seamless data access as well as support for big data processing services to DestinE users. One of the big data processing services offered via DEDL is based on the Dask ecosystem. The DEDL Dask service allows for multi-cloud processing, following the paradigm of data proximate computation. Digital Twins of the Earth will run on HPC system across Europe, producing petabytes of data per day over the next few years. Accordingly, enabling users to do processing close to the data is key to cope with these volumes of data in an efficient way. Those capabilities of the DEDL are presented by two use case demonstrations based on Pangeo core packages such as xarray and Dask.