Dynamic Batch Sizing for Distributed Data Stream Processing Systems

In the data stream pipelines, variations of data stream ingestion rate and static batching are main challenges for processing data streams in real time.When the data stream ingestion rate and batch interval time are high, many events may be pushed into the channel, potentially exceeding transaction capacity, exhausting the agent's memory, and increased transmission time, and longer event queuing times in the channel.In this paper, a dynamic batch interval control algorithm is proposed to handle bursts in data volume, balancing the complex relationship between throughput, transmission time, and channel availability.The proposed method is implemented in Apache Flume for data ingestion and apply Spark Streaming for processing.The experiments indicate that the dynamic batching can prevent memory channel failures, reduce transmission time in data ingestion and processing time in processing.Moreover, the proposed approach minimizes the event queuing time in channel and maximizes the throughput with adjustable processing time for both ingestion and processing phases compared to static batching.