Multi-Region Cloud-Native LLM Inference and Dynamic KV-Cache Routing

Serving large language models (LLMs) across multi-region cloud infrastructure introduces severe memory bandwidth bottlenecks and unpredictable tail latencies. During autoregressive generation, each active request maintains a growing key-value (KV) state tensor in high-bandwidth GPU memory (HBM). When stateless cloud load balancers distribute incoming requests using round-robin or least-connections policies, identical system prompts, multi-turn conversation histories, and shared retrieval contexts are repeatedly recomputed across separate GPU nodes. This paper presents a cloud-native, prefix-aware KV-cache routing and disaggregated inference architecture designed for multi-region Kubernetes clusters. Building on PagedAttention virtual memory management (Kwon et al., 2023), FlashAttention IO-aware kernels (Dao et al., 2022), and disaggregated prefill-decode scheduling (Zhong et al., 2024), the proposed architecture integrates a consistent-hashing radix tree directory at the global edge ingress layer with a three-tier memory hierarchy spanning GPU HBM, host DDR5 DRAM, and NVMe-over-Fabrics (NVMe-oF) flash pools. We evaluate the architecture through a detailed production case study of Anyscale Ray Serve and vLLM deployments operating across three cloud regions, alongside Cloudflare Workers AI edge routing patterns. Across a 72-hour production trace on 64 NVIDIA H100 GPUs serving 70-billion-parameter models, dynamic prefix-tree routing increases global KV-cache hit rates from 6.4% to 82.3%, reduces median time-to-first-token (TTFT) by 67.9% (from 420 ms to 135 ms), cuts 99th-percentile TTFT by 78.3% (from 1,890 ms to 410 ms), and improves sustainable throughput per GPU by 210.7%. Finally, we analyze failure recovery under availability-zone partitions, cross-tenant side-channel risks in shared prefix caches, and FinOps cost per million generated tokens. Research Domain & Specialization Track Master of Science in Information Technology (MSIT) — Distributed AI Systems and Cloud-Native Infrastructure. Core Technical Highlights Empirical Production Case Studies: Rigorous architectural blueprints and telemetry evaluation across enterprise cloud deployments. Four-Tier Reference Architecture: Formal specification from edge ingress to distributed hardware execution pools. Mathematical Formulation: Formally derived algorithmic and optimization models with closed-loop verification. Multi-Modal Benchmarking: Comparative empirical baselines under fault-injection and synthetic workload scaling. Author Information Author: Sonu Kumar SinghRole: Senior Consultant Affiliation: Independent Researcher Publication Status: Preprint / Working Paper (MSIT AI & Cloud Systems Research Series) Reproducibility & Open Science All included architecture diagrams, telemetry schemas, and benchmark curves are open-access under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.