ABSTRACT Time series data flows through a four-stage pipeline: outlier removal (autoregressivefilter), fault removal (rangefilter), gap filling (dailyₚattern), and smoothing (gaussianₖernel). Water resource recovery facilities generate tremendous amounts of process data, most of it in the form of time series. This data can be found in supervisory control and data acquisition systems, process historians, or relational databases where metadata is (hopefully) abundant. Removal of data artefacts and sensor faults, as well as smoothing and aggregation, are all essential steps in transforming raw data into a form fit for modelling, analysis, and decision-making. These transformations typically ignore metadata, meaning metadata-rich datasets can quickly lose all context, traceability, and reusability. The context at risk of being lost includes both the original metadata and descriptions of the processing itself. Failure to report this metadata is due to the difficulty of manual reporting and to the sometimes highly iterative nature of data pre-processing. This paper presents metEAUdata, a framework that automates metadata creation and preservation during pre-processing operations. Unlike existing solutions that bundle limited preprocessing algorithms, metEAUdata provides a lightweight interface enabling any algorithm to become metadata-aware through simple wrapper functions, extending metadata recording to existing scientific libraries with minimal effort. By automating the creation of structured, extensive metadata for arbitrary preprocessing pipelines, metEAUdata supports the transition towards FAIR-compliant time series data.
Therrien et al. (Fri,) studied this question.