Optimizing fault tolerance in resource-efficient distributed stream processing

Distributed Stream Processing (DSP) focuses on the near real-time processing of unbounded data streams, often under strict Quality of Service requirements. In many scenarios, users require results to be consistent even in the presence of partial failures, which necessitates exactly-once processing guarantees and automated recovery mechanisms. At the same time, stream processing workloads commonly change over time, making static resource configurations inefficient, while over-provisioned setups can be costly from both economic and environmental perspectives. Maintaining continuous low-latency operation and timely failure recovery in such dynamic environments—without resorting to wasteful over-provisioning—presents a central challenge. In this thesis, we introduce a recovery time constraint to guide the automatic configuration of DSP jobs, specifying how quickly the system must resume processing near the current event timestamp after a failure. We begin by focusing on a single parameter, the checkpoint interval, for workloads with stable throughput rates, modeling snapshot overhead alongside the requirement for reliable post-failure catch-up. Short-lived profiling is used to collect performance metrics, coupled with a heuristic for estimating recovery times under static conditions. Next, we extend this idea to dynamic resource scaling for workloads that vary over time. By leveraging time series forecasting to anticipate changes in incoming data rates, we adapt the recovery time heuristic and proactively allocate resources, ensuring the constraint remains satisfied. Finally, we combine these ideas into a multi-parameter optimization method, employing multi-objective Bayesian search to jointly tune checkpoint intervals, parallelism, CPU cores, and memory allocations for different workload scenarios. Building on the previous approaches, this method continuously refines performance models to identify configurations that minimize average latencies and resource usage while consistently meeting user-defined recovery targets. All approaches were prototyped using Apache Flink and deployed on commodity Kubernetes clusters, evaluating them on widely used benchmark applications under conditions that closely mirror real-world operational environments. Empirical results confirm that explicitly optimizing the recovery time constraint yields reliable failure recovery, high performance, and lower operational costs—while consistently preserving exactly-once semantics under changing workloads. Overall, this thesis provides novel methods for fault-tolerant DSP deployments, ensuring low latencies and predictable recovery times without resorting to resource over-provisioning.