RLCharge: Imitative Multi-Agent Spatiotemporal Reinforcement Learning for Electric Vehicle Charging Station Recommendation

Electric Vehicle (EV) has become preferable choices in modern transportation system due to its environmental and energy sustainability. However, in many large cities, EV drivers often fail to find proper spots for charging because of the limited charging infrastructures and spatiotemporally unbalanced charging demands. Indeed, the recent emergence of deep reinforcement learning provides great potential to improve charging experience over long-term horizons. In this paper, we propose RLCharge for intelligent EV charging station recommendation by jointly considering various long-term spatiotemporal factors. Specifically, by regarding each charging station as an agent, we formulate the problem as a multi-objective multi-agent reinforcement learning task. We first develop a multi-agent actor-critic framework with centralized training decentralized execution. Particularly, we propose a tailor designed centralized attentive critic with the delayed access strategy to coordinate the recommendation between geo-distributed agents during centralized training. Besides, we propose the spatio-temporal heterogeneous graph convolution module to handle the partial observability problem during decentralized execution. After that, to effectively optimize multiple divergent objectives, we develop a dynamic gradient re-weighting strategy to adaptively guide the optimization direction, and propose an adaptive imitation learning scheme to further accelerate and stabilize the policy convergence. Finally, extensive experiments on two real-world datasets demonstrate that RLCHARGE achieves the best comprehensive performance compared with ten baseline approaches.