This paper explores the role of next-generation data pipeline automation in enhancing the efficiency and scalability of business intelligence (BI) systems. With the increasing volume and complexity of data, traditional manual extraction, transformation, and loading (ETL) processes have proven inadequate to meet the demands of modern BI applications. Automation offers a transformative solution, enabling faster, more reliable, and scalable data processing workflows. This paper introduces a conceptual framework for data pipeline automation that incorporates microservices, real-time data processing, and cloud-native technologies such as serverless computing and containerization. Through detailed analysis of core components, tools, and technologies like Apache Airflow, dbt, and Kafka, we highlight how automation streamlines BI workflows, improves data quality, and ensures real-time responsiveness. The paper also examines challenges such as latency, data governance, and security, offering insights into overcoming these barriers. Furthermore, we explore future research directions, including AI-driven operations, no-code platforms, and multi-cloud architectures, which promise to revolutionize automated pipeline management further. This study provides valuable insights for both researchers and practitioners aiming to optimize data pipelines for modern BI systems.
No takes yet. Share an insight, caveat, or question.
Ogeawuchi et al. (2022) studied this question.