Reinforcement Learning for Cyber-Physical Systems

Cyber-Physical Systems (CPS), including smart industrial manufacturing, smart transportation, and smart grids, among others, are envisioned to convert traditionally isolated automated critical systems into modern interconnected intelligent systems via interconnected human, system, and physical assets, as well as providing significant economic and societal benefits. The characteristics of CPS include complexity, dynamic variability, and heterogeneity, arising from interactions between cyber and physical subsystems. These characteristics introduce critical challenges in addition to existing and vital safety and reliability requirements from traditional critical systems. To overcome these challenges, Artificial Intelligence (AI) and Machine Learning (ML) schemes, which have proven effective in numerous fields (robotics, automation, prediction, etc.), can be leveraged as solutions for CPS. In particular, reinforcement learning can make precise decisions automatically to maximize cumulative reward via systematic trial and error in an unknown environment. Yet, challenges still remain for integrating complex reinforcement learning systems with dynamic and diverse CPS domains. In this paper, we conduct a thorough investigation of existing research on reinforcement learning for CPS, and propose a framework for future research. In addition, we carry out two case studies on reinforcement learning in transportation CPS and industrial CPS to validate the effectiveness of reinforcement learning in targeted applications. Using realistic simulation platforms, we validate the effectiveness of reinforcement learning for decision making in routing for transportation CPS and production control for industrial CPS. Finally, we outline some future research challenges that remain.