Quality Alerting System for Data Analysis in Data Lakes

Data lakes have become increasingly popular as centralized repositories for storing large volumes of heterogeneous and unstructured data. However, the sheer amount of data in data lakes makes it challenging to ensure its quality. Data quality can lead to accurate analyses and decisions that benefit an organization. Therefore, there is a need for a proper quality alert system in data lakes that can identify data quality issues and notify users in real-time. This research presents a novel quality alert system integrated within the data lake architecture, enabling real-time identification and notification of data quality issues during analysis. Based on a quality model and an alert process, the proposed alert system enables real-time identification and notification of data quality issues during analysis. This should reduce the costs associated with data integration. In addition, the alert system may measure the potential biases that could arise from analyzing poor-quality data, allowing users to make informed decisions based on the quality of the data.