Answers are grounded in this paper and cite their sources.
Enterprise data platforms are increasingly required to process massive volumes of heterogeneous and continuously generated data while simultaneously delivering faster analytics, predictable performance, operational reliability, and sustainable cloud economics. Cloud infrastructure has substantially expanded the computational capacity available to data engineering teams, but elastic resources alone cannot resolve the architectural limitations created by inefficient pipelines, tightly coupled storage and processing, poorly optimized data structures, or fragmented serving environments. Enterprise data engineering therefore faces a fundamental tension between the necessity for massive computational scale and the operational requirement for efficient, tightly controlled pipelines that prevent runaway cloud costs and system bottlenecks. This study develops a cloud-native architectural framework for enterprise-scale big data platforms that addresses this tension by treating scalability and optimization as complementary capabilities rather than competing design objectives. The proposed architecture is organized around three interconnected layers: an elastic data foundation based on the decoupling of compute and storage; a distributed processing layer combining parallel computation with automated and intelligent pipeline engineering; and a high-performance serving layer designed for database-level optimization and low-latency analytical access. The framework extends these architectural layers through intelligent data engineering capabilities involving workload orchestration, incremental processing, pipeline observability, metadata management, data quality, lineage, governance, and cost-aware resource allocation. A central argument of the study is that enterprise scalability should not be measured simply by the amount of computational capacity a platform can provision. Instead, it should be evaluated by the platform's ability to allocate computational resources intelligently while minimizing unnecessary data movement, repeated processing, operational bottlenecks, and persistent infrastructure costs. Accordingly, the framework introduces Intelligent Data Engineering as the coordinating capability through which storage, distributed computation, pipeline automation, and analytical serving are transformed into an adaptive enterprise data platform. Rather than applying increasing computational resources to inefficient workloads, the architecture separates where different forms of scalability, processing, and optimization should occur. The resulting model supports batch, streaming, and large-scale incremental workloads while maintaining a distinct high-performance serving environment for analytics and downstream intelligent applications. It also establishes a foundation for enterprise artificial intelligence by enabling governed, observable, and reusable data assets to move efficiently from raw ingestion to analytical consumption. The study ultimately argues that the objective of cloud-native big data architecture is neither maximum scale nor maximum local optimization in isolation. It is to construct an intelligent platform in which elastic capacity becomes available when computational intensity requires it, while architectural discipline ensures that performance, reliability, and economic efficiency are preserved as the enterprise data estate grows.
No takes yet. Share an insight, caveat, or question.
Sai Prakhyath Jawaji (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: