Exploratory study of the ELK stack for meteorological observation system data analysis

Different kinds of sensors compose a meteorological observation system that measures meteorological variables. Sensors can collect data for a long period of time in a high sampling frequency. Some meteorological parameters can be determined by making measurements that ranges from a few seconds to annual measurements which depends on the kind of equipment and application needs. In this scenario, data management is not a trivial task due to heterogeneity, large amount of data and also to the usage of proprietary software for data gathering and handling. We used a data acquisition system (datalogger) to collect and store data from a thermo-baro-hygrometer, and a pyranometer, which were calibrated previously in the laboratory. This paper aimed to analyze the open source Elasticsearch, Logstash and Kibana (ELK) stack to capture, transform, enrich, store, index, select relevant time slots and generate graphs that were integrated in a dashboard for combined visualization and analysis. Additionally, we explored its capacity to embed metadata from sensors and correct data based on a calibration certificate, also showing some relevant graphics. In this weather application, we observed that this set of computational tools are well suited to manage the daily difficulties in handling meteorological data and metadata.