Data Quality and Incremental Data Loading in ETL Pipelines: A Comparative Survey of Industry Practices and Student Readiness

Extract–Transform–Load (ETL) pipelines depend on data quality validation, reconciliation and correct identification of changed records. This cross-sectional questionnaire survey compares practices reported by working professionals with the awareness and practical exposure of IT students. Of 90 submitted responses, 84 were analysed (34 professionals, 50 students) after excluding non-consenting and unclassifiable responses. Professionals most often reported null checks, format validation and duplicate detection, and incremental loading was the most common primary approach. Students reported lower familiarity with incremental loading, data reconciliation and Change Data Capture (CDC) and limited project-based exposure. Professionals reported higher ETL familiarity than students (Mann–Whitney U = 1179, p = 0.00164). The study is exploratory and relies on self-reported data from a convenience sample, so its findings should not be generalised