This paper presents an AI-driven autonomous optimization framework for Apache Kafka deployed on AWS Managed Streaming for Apache Kafka (MSK), targeting high-volume financial systems such as home lending, auto lending, payments, and fraud detection infrastructure. Modern financial institutions rely on event-driven streaming architectures to process millions of transactions per hour. However, default Kafka configurations frequently prove insufficient for burst-heavy workloads, leading to latency spikes, partition imbalances, and cascading processing delays. This research addresses these challenges by formulating Kafka configuration tuning as a Markov Decision Process (MDP) and applying a reinforcement learning (RL) agent to dynamically optimize parameters including batch size, I/O thread concurrency, partition count, and consumer fetch settings in real time. The proposed closed-loop architecture integrates continuous telemetry collection via Amazon CloudWatch and Kafka JMX, stateful feature engineering, an RL-based decision engine, and automated configuration execution. Experimental evaluation on an MSK cluster simulating financial workloads demonstrates up to 35% throughput improvement and 28% latency reduction compared to static baseline configurations, along with improved resilience during burst traffic scenarios. This work also discusses the broader economic implications of optimized streaming infrastructure for U.S. financial system stability, systemic risk mitigation, and digital banking modernization. Status: Preprint — under review at The VLDB Journal (Manuscript No. VLDB-D-26-00067, Springer).
Bibek Maharjan (Fri,) studied this question.