Building a Resilient and Self-Healing ELT Pipeline Using Metadata-Driven Architecture

Modern organizations depend on Extract, Transform, and Load (ETL) pipelines to process large volumes of data for analytics and decision-making. However, traditional ETL pipelines are often tightly coupled, difficult to maintain, and highly dependent on manual intervention when data quality failures occur. Common issues such as missing values, duplicate records, invalid data ranges, and schema inconsistencies can disrupt pipeline execution and reduce data reliability. To address these challenges, this research presents a metadata-driven self-healing ETL pipeline that automatically detects and recovers from common data quality failures using externally defined metadata rules. The proposed framework separates validation and recovery logic from application code through a centralized YAML-based metadata layer. The architecture consists of ingestion, transformation, validation, self-healing, and logging modules that work together to improve pipeline resilience and maintainability. Validation rules are dynamically loaded from metadata configurations, while automated recovery mechanisms perform data correction, deduplication, and failure handling without requiring manual intervention. The system was implemented using Python and evaluated on a production-style dataset containing intentionally introduced data quality failures. Experimental results demonstrate that the proposed approach successfully eliminated missing values, duplicate records, and invalid numeric values while maintaining data consistency and improving overall data quality. The findings indicate that metadata-driven architectures can significantly enhance the reliability, flexibility, and operational efficiency of modern ETL pipelines.