Enhancing Data Cleaning through the Extraction and Expansion of Relaxed Functional Dependencies

Data cleaning is pivotal for ensuring high-quality datasets in machine learning and data analytics, yet traditional methods often rely on manual thresholding, which is subjective and inefficient. This paper proposes RFD-Sugeno-KMeans-Clean, a data cleaning framework that integrates a Sugeno fuzzy inference system for automatic thresholding to extract Relaxed Functional Dependencies (RFDs). Then, by combining K-means++ clustering, it extracts the most effective pattern functional dependencies for data cleaning from complex datasets. In the second phase, a two-step SQL method is employed to detect and clean the data. The method also utilizes Principal Component Analysis (PCA), Identifying and Repairing Multiple Violations (Spline), Median, and Adaptive Neuro-Fuzzy Inference System (ANFIS) to enhance outlier detection across diverse datasets. The effectiveness of the proposed method has been validated using a dataset from the UCI database. The results demonstrate that the proposed method achieves a significant reduction in the number of data points and tuples, with a decrease of approximately 99.5% in most cases, while ensuring stability and reliability in responses. The proposed method’s ability to extract optimal patterns in data cleaning demonstrates higher stability and reliability in responses compared to earlier methods, making it a viable solution for addressing data cleaning challenges in various domains.