Conjectural Online Learning with First-order Beliefs in Asymmetric Information Stochastic Games

Asymmetric information stochastic games (aisgs) arise in many complex socio-technical systems, such as cyber-physical systems and IT infrastructures. Existing computational methods for aisgs are primarily offline and can not adapt to equilibrium deviations. Further, current methods are limited to special classes of aisgs to avoid belief hierarchies. To address these limitations, we propose conjectural online learning (col), an online method for generic aisgs. col uses a forecaster-actor-critic (fac) architecture where subjective forecasts is used to conjecture the opponents' strategies and break belief hierarchies (forecaster), online rollout is used to adapt strategies to nonstationary environments (actor), Monte-Carlo simulation is used to estimate costs (critic), and Bayesian learning is used to calibrate conjectures. We prove that the conjectures produced by col are asymptotically consistent with the information feedback in the sense of a relaxed Bayesian consistency. We also prove that the empirical strategy profile induced by col converges to the Berk-Nash equilibrium, a solution concept characterizing rationality under subjectivity. Experimental results from an intrusion response use case demonstrate col's superiority over state-of-the-art reinforcement learning methods against nonstationary attacks.