Optimizing Real-Time Big Data Pipelines: A Comparative Analysis of Edge Computing vs. Cloud Architectures

The exponential growth of Internet of Things (IoT) devices has created a paradigm shift in data processing, moving from centralized cloud architectures toward decentralized edge computing. This research investigates the performance optimization of real-time big data pipelines by comparing traditional cloud-based processing with an edge-cloud hybrid model. Utilizing a discrete-event simulation methodology, the study analyses 1,000,000 simulated smart city data events to measure two critical metrics: end-to-end latency and network bandwidth consumption. The results demonstrate that the edge-cloud hybrid model reduces average processing latency by 75%, from 180 ms to 45 ms, while simultaneously decreasing bandwidth usage by approximately 77% through localized data filtering. The findings suggest that while edge computing significantly enhances responsiveness for time-sensitive applications—such as autonomous systems and real-time healthcare monitoring—it is most effective when integrated into a hybrid framework that leverages the cloud for long-term storage and heavy computational modelling. This paper concludes that decentralized architectures are essential for scaling the next generation of big data infrastructure, providing a roadmap for future research into localized AI decision-making.