AI-Augmented Compliance Auditing for Cloud Systems: A Hybrid ML–LLM Approach

Manual compliance auditing in cloud environments consumes up to 40% of IT security budgets annually, yet existing approaches verify control presence rather than effectiveness, leaving institutions vulnerable to adversarial evasion. This paper presents an AI-augmented hybrid ML–LLM compliance auditing system evaluated on Rwanda’s National Cyber Security Authority (NCSA) Minimum Cybersecurity Standards (169 controls across 14 families). The system combines leakage-free XGBoost multi-label classification with GPT-4o-mini semantic log analysis, grounded in a formal effectiveness model. Key findings: (1) XGBoost v2 achieves 85.45% macro-F1 on leakage-free synthetic data (Wilson 95% CI = [84.9%, 86.0%]); an initial 86.3% data-leakage rate artificially inflated prior results to 99.99% and was identified and corrected in this revision; (2) GPT-4o-mini achieves 92.3% macro accuracy across four log types (n = 628, 37.5% real enterprise data, Wilson CI = [89.9%, 94.3%]); (3) adversarial validation across five MITRE ATT&CK scenarios yields 92.8% macro detection with 0.0% false-positive rate on real SSH/PAM compliant logs (n = 75); (4) a cross-dataset generalization analysis confirms 87.6% F1 on real SSH logs but identifies a 37.8-percentage-point out-of-vocabulary gap for Windows and HTTP log types, motivating the hybrid architecture; (5) the combined hybrid system (XGBoost for in-vocabulary logs, GPT-4o-mini for out-of-vocabulary) achieves 85.1% F1 with 6.4% false-positive rate on 180 real-world logs. The system runs at 2.0 CPU cores, 2.66 GB RAM, on $50/month cloud hosting (Apple M1 Pro baseline; storage and maintenance excluded), producing audit reports in 2–5 s depending on log volume and policy document size, demonstrating that effectiveness-based compliance auditing is accessible without enterprise-grade infrastructure.