Machine Learning‐Based Data Deduplication: Techniques, Challenges, and Future Directions

ABSTRACT Data deduplication plays an important role in modern data management as it reduces storage costs and ensures consistency by eliminating redundant records. The traditional data deduplication methods are effective for exact matches but struggle with adaptability and detecting near‐exact duplicate records in unstructured or complex data. Machine learning (ML) addresses these limitations by using pattern recognition, feature learning, and statistical modeling to identify subtle similarities between records. This review classifies ML‐based deduplication techniques into supervised, unsupervised, semi‐supervised, and deep learning methodologies. It also discusses key challenges, including class imbalance, model interpretability, and computational overhead. The paper also explores recent developments in federated learning, real‐time deduplication, and multimodal techniques to highlight current trends in these areas. Finally, the paper identifies key open issues and proposes a unified perspective for scalable, real‐time deduplication systems that can accommodate diverse data types, structures, and system requirements.