Bringing computation to the data: A MOEA-driven approach for optimising data processing in the context of the SKA and SRCNet

The Square Kilometre Array (SKA) is a next-generation radio astronomy-driven big data facility that will revolutionise our understanding of the Universe and the laws of fundamental physics, and needs innovative solutions for efficient data processing. The SKA Regional Centres Network (SRCNet) is a collaborative ecosystem tasked with the demanding role of processing and analyzing SKA data products. With SKA, the near-exascale computing will be a challenge, chief among them being the issue of data movement. As computational capabilities, the sheer volume of generated data becomes staggering. The traditional approach of moving tons of data to centralized computing resources becomes impractical due to the limitations of existing networks and storage infrastructures. The data transfer bottleneck becomes a critical impediment, hindering the overall efficiency. To overcome this challenge, a paradigm shift is imperative. Strategies such as in-situ processing and distributed computing models where computation is moved to the data emerge as promising solutions. In the realm of SKA and specifically within SRCNet data processing needs, the conjunction of Function-as-a-Service (FaaS) with a decision-making entity driven by Evolutionary Algorithms (EAs) becomes pivotal. FaaS abstracts away infrastructure management concerns, enabling the deployment of modular functions in close proximity to data sources. This development aligns with the principle of bringing computation to the data, mitigating the challenges associated with extensive data transfers. The decision-making entity, guided by EAs, facilitates a systematic exploration of near-optimal execution plans, that will provide with detailed information on how and where a function should be executed within the overall computing and data infrastructure. With the focus on two objectives such as execution time and energy consumption, and constraints like data transfers or data locations, Multi-Objective Evolutionary Algorithms (MOEAs) could be a good option to optimise the movement of the computation to the data. In this context, MOEAs could provide a baseline guide for efficient and cost-effective data processing for the computation model within the SRCNet. Our proposal unifies the technical aspects of FaaS deployment, together with the mathematical modelling and code implementation of a customised first approach MOEA model for the optimisation of function execution plans within the SRCNet architecture and its integration with FaaS.