Large Language Models for Intelligent Data Engineering: Automating Schema Design, Lineage, and Quality Control

This research investigates the application of large language models (LLMs) to automate key data engineering tasks, including schema design, data lineage tracking, and quality control. It presents an integrated framework combining language models with traditional data engineering tools to improve productivity while maintaining accuracy and reliability. The paper explores automated schema generation from business requirements, lineage extraction from SQL queries and ETL scripts, and AI-enhanced data quality monitoring. It also discusses validation techniques, human-in-the-loop review, prompt engineering, error handling, and integration with existing engineering workflows. The study reports potential reductions of approximately 70% in schema design time, improvements of over 80% in lineage documentation completeness, and a 65% acceleration in quality issue detection. It also addresses limitations such as hallucinations, inconsistent outputs, context-window constraints, and privacy risks. This work provides practical guidance for organizations adopting LLM-based data engineering automation while retaining essential human oversight and production validation.