BEACON - A high-performance data lake solution

In many of the societal and scientific challenges, such as Digital Twins of the Oceans and virtual research environments, fast access to a large number of multidisciplinary data resources is key. However, achieving performance is a major challenge as original data is in many cases organised in millions of observation files which makes it hard to achieve fast responses. Next to this, data from different domains are stored in a large variety of data infrastructures, each with their own data-access mechanisms, which causes researchers to spend much time on trying to access relevant data. In a perfect world, users should be able to retrieve data in a uniform way from different data infrastructures following their selection criteria, including for example spatial or temporal boundaries, parameter types, depth ranges and other filters. Therefore, as part of the EOSC Future and Blue-Cloud projects, MARIS has started developing a software system called ‘BEACON’ with a unique indexing system that can, on the fly with high performance, extract specific data based on the user’s request from millions of observational data files, containing multiple parameters in diverse units. The BEACON system and its data can be accessed via a REST API that is exposed by BEACON itself, meaning clients can query data via a simple JSON request. The system is built in a way that it returns one single harmonised file as output, regardless of whether the input contains many different data types or dimensions. It also allows for converting the units of the original data if parameters are measured in different types of units (for this it e.g. makes use of the NERC Vocabulary Service and the I-Adopt framework). It is important to mention that the system can be applied to different data infrastructures and is not tailor made for one specific type of database.  Showcasing the performance and usability of BEACON, as part of this presentation, the BEACON system will be shown with practical example applications on the SeaDataNet CDI database, Euro-ARGO database and the ERA5 dataset from the Climate Data Store. It will be explained how we want to create an open community around BEACON, how the BEACON software could be accessed and used by developers and the most recent developments will be shown. More technical details, example applications and general information for BEACON can be found on the website https://beacon.maris.nl/.