The rapid growth of cloud-native architectures has transformed how organizations manage and operationalize data. In this context, automating data pipelines has become critical for ensuring agility, scalability, and efficiency in handling large-scale, dynamic datasets. This paper proposes a conceptual framework for automating data pipelines using Extract-Load-Transform (ELT) tools in cloud-native environments. Traditional ETL approaches often struggle with the volume and velocity of modern data workloads, whereas ELT strategies leverage the scalable compute power of cloud services to transform data post-loading, thereby enhancing performance and flexibility. The proposed framework integrates key cloud-native principles—such as containerization, microservices, serverless computing, and infrastructure as code—to orchestrate the data lifecycle from ingestion to insight. It emphasizes modular pipeline design, metadata-driven orchestration, dynamic schema evolution, and real-time monitoring to achieve seamless automation. Critical considerations such as data quality assurance, governance, security, and cost optimization are embedded throughout the pipeline's architecture to meet enterprise demands. By utilizing tools such as Apache Airflow, dbt, Fivetran, and cloud-specific services like AWS Glue and Google Cloud Dataflow, the framework enables automated, scalable, and resilient data operations. The paper also discusses the importance of decoupling compute and storage, employing containerized workloads for pipeline tasks, and leveraging managed orchestration services to reduce operational overhead. Case studies highlight the framework’s effectiveness in accelerating data delivery, improving reproducibility, and lowering time-to-insight across diverse industries. Furthermore, the framework addresses the challenges associated with multi-cloud strategies, hybrid architectures, and evolving data compliance requirements. Future research directions include integrating AI-driven optimizations for dynamic workload balancing, adaptive pipeline healing, and intelligent schema inference. Overall, this conceptual framework provides a robust foundation for enterprises seeking to modernize their data infrastructure and harness the full potential of cloud-native ecosystems. It lays the groundwork for continuous innovation in data engineering practices, positioning automated ELT pipelines as a cornerstone for next-generation analytics, machine learning, and digital transformation initiatives.
Akindemowo et al. (Fri,) studied this question.