Cost-Aware Optimization of Cloud-Native Data Pipelines Using Adaptive Scheduling and Query Planning

Cloud-native data engineering platforms such as BigQuery, Snowflake, and Apache Spark provide scalable infrastructure but often incur unpredictable operational costs. This paper presents a cost-aware optimization framework that integrates execution metadata, workload characteristics, and SLA constraints to dynamically optimize scheduling and compute allocation. Experimental evaluation demonstrates average cost savings between 18–27%, with peak reductions up to 37%, while maintaining SLA compliance with minimal latency overhead. The proposed system introduces a vendor-agnostic control layer for adaptive optimization without modifying underlying execution engines. This work presents an applied systems approach to cost-aware optimization in cloud-native data engineering pipelines, based on experimental evaluation of representative analytical workloads.