Conjectural Online Learning with First-order Beliefs in Asymmetric Information Stochastic Games

Asymmetric information stochastic games ({aisg}s) arise in many complex socio-technical systems, such as cyber-physical systems and IT infrastructures. Existing computational methods for {aisg}s are primarily offline and can not adapt to equilibrium deviations. Further, current methods are limited to special classes of {aisg}s to avoid belief hierarchies. To address these limitations, we propose conjectural online learning ({col}), an online method for generic {aisg}s. {col} uses a forecaster-actor-critic ({fac}) architecture where subjective forecasts is used to conjecture the opponents' strategies and break belief hierarchies (forecaster), online rollout is used to adapt strategies to nonstationary environments (actor), Monte-Carlo simulation is used to estimate costs (critic), and Bayesian learning is used to calibrate conjectures. We prove that the conjectures produced by {col} are asymptotically consistent with the information feedback in the sense of a relaxed Bayesian consistency. We also prove that the empirical strategy profile induced by {col} converges to the Berk-Nash equilibrium, a solution concept characterizing rationality under subjectivity. Experimental results from an intrusion response use case demonstrate {col}'s superiority over state-of-the-art reinforcement learning methods against nonstationary attacks.