Chaos Engineering and Automated Fault-Injection Verification for Distributed Cloud AI Microservices

Modern enterprise artificial intelligence platforms decompose complex inference workflows—spanning API gateways, feature stores, vector databases, guardrails, and multi-GPU model serving pods—into deeply nested cloud-native microservices. While microservice disaggregation enables independent horizontal scaling across heterogeneous hardware accelerators, it simultaneously expands the combinatorial state space of partial failures, including GPU High Bandwidth Memory (HBM) exhaustion, NCCL collective communication hangs, NVMe thermal throttling, and cascading retry storms. Conventional static unit tests and synthetic staging benchmarks fail to expose these emergent failure modes before production customer traffic is impacted. This paper presents an automated Chaos Engineering and Fault-Injection Verification framework tailored for distributed cloud AI microservices running on Kubernetes and Istio service meshes. Integrating principles from the Netflix Chaos Automation Platform (ChAP) and Gremlin enterprise fault injection on Amazon Elastic Kubernetes Service (EKS), the proposed architecture orchestrates blast-radius-isolated canary experiments using HTTP/gRPC request-header routing, eBPF kernel-level hardware fault emulation, and Sequential Probability Ratio Test (SPRT) automated abort guardrails. We evaluate the verification platform across a 48-microservice generative AI and recommendation inference cluster subjected to 140 automated chaos campaigns over a 90-day production cycle. Empirical results demonstrate that continuous fault-injection verification uncovered 27 latent cascading failure defects prior to general availability, reduced Mean Time to Detection (MTTD) by 76.8% (from 18.5 minutes to 4.3 minutes), compressed Mean Time to Recovery (MTTR) by 81.4% (from 47.2 minutes to 8.8 minutes), and improved steady-state p99 SLO compliance during real AWS availability zone perturbations from 94.10% to 99.96%. Research Domain & Specialization Track Master of Science in Information Technology (MSIT) — Resilience Engineering, Fault Tolerance, and Cloud Verification. Core Technical Highlights Empirical Production Case Studies: Rigorous architectural blueprints and telemetry evaluation across enterprise cloud deployments. Four-Tier Reference Architecture: Formal specification from edge ingress to distributed hardware execution pools. Mathematical Formulation: Formally derived algorithmic and optimization models with closed-loop verification. Multi-Modal Benchmarking: Comparative empirical baselines under fault-injection and synthetic workload scaling. Author Information Author: Sonu Kumar SinghRole: Senior Consultant Affiliation: Independent Researcher Publication Status: Preprint / Working Paper (MSIT AI & Cloud Systems Research Series) Reproducibility & Open Science All included architecture diagrams, telemetry schemas, and benchmark curves are open-access under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.