Optimizing Big Data Workloads in Hybrid Cloud Environments: Evaluating File Formats and Compression Codecs

Hybrid cloud architectures offer a practical way for enterprises to handle peak big data workloads by extending on-premise systems into the public cloud and bursting workloads there when needed. However, limited network bandwidth between the two environments can become a performance bottleneck. This paper investigates the influence of various file formats and compression codecs on query execution time and data transfer in bandwidth-constrained hybrid clouds, utilizing the TPC-DS benchmark and Apache Spark. We introduce new performance metrics and simulate realistic hybrid cloud conditions using traffic shaping. In our experiments, execution time degradation reached 233.81% for uncompressed data compared to the baseline on-premise workload execution. With Gzip compression, this was reduced to 134.91%.