Addressing Extreme Baseline Imbalances in Quasi-Experimental Evaluation of AI-Driven Adaptive Cybersecurity Training: A Multi-Method Approach

Despite widespread adoption of cybersecurity awareness training (CSAT), a persistent knowledge–behaviour gap continues to undermine organisational security posture, particularly in resource-constrained and developing-country contexts. This 12-week quasi-experimental field study evaluated an AI-adaptive CSAT platform against traditional instructor-led training (ILT) across three Yemeni organisations (total N = 187; AI-Adaptive: n = 94; Control: n = 93). The system used a 4-parameter Bayesian Knowledge Tracing (BKT) engine—with interpretable guess and slip signals—as an auditable pedagogical decision layer that triggered Protection Motivation Theory (PMT) and Theory of Planned Behavior (TPB)-aligned interventions. Extreme baseline imbalances (Cohen’s d > 2.0), at which standard ANCOVA residual adjustment alone is known to be biased and which necessitated advanced causal-inference triangulation, were addressed via a four-method protocol (ANCOVA, Propensity Score Matching, Difference-in-Differences, mixed-effects). All four methods converged on consensus effect sizes of d = 0.66–0.89. IT-verified Tier 2–3 incidents declined by 48.9% (incidence-rate ratio [IRR] = 0.51, 95% CI [0.38, 0.68]); blinded phishing click-rates fell from 8.8% to 2.1% (χ2(1) = 8.74, p = 0.003). Bootstrapped mediation analysis (PROCESS Model 4; 5000 draws) indicated that coping self-efficacy and perceived behavioural control—but not threat appraisal—were jointly associated with 66.4% of the total compliance effect. Rosenbaum bounds Γ = 2.1; E-values ≥ 3.4. The findings are consistent with the hypothesis that AI-adaptive cybersecurity training produces robust, theoretically explicable benefits and that the coping-appraisal pathway, not threat salience, is the active psychological mechanism. The four-method triangulation framework offers a replicable standard for field evaluations with non-random assignment. the consensus envelope d = 0.66–0.89 is the observed range of point estimates across the four estimators; per-method 95% CIs are reported below indirect effect via coping self-efficacy = 0.843 [0.52, 1.19], via PBC = 0.524 [0.28, 0.81], via threat appraisal = 0.059 [−0.07, 0.21] (ns); direct effect c’ = 0.63 (p = 0.026); total effect c = 2.06 [1.58, 2.54].