Learning to overtake in TORCS using simple reinforcement learning

In modern racing games programming non-player characters with believable and sophisticated behaviors is getting increasingly challenging. Recently, several works in the literature suggested that computational intelligence can provide effective solutions to support the development process of NPCs. In this paper, we applied the Behavior Analysis and Training (BAT) methodology to define a behavior-based architecture for the NPCs in The Open Racing Car Simulator (TORCS), a well-known open source racing game. Then, we focused on two major overtaking behaviors: (i) the overtaking of a fast opponent either on a straight stretch or on a large bend; (ii) the overtaking on a tight bend, which typically requires a rather advanced braking policy. We applied simple reinforcement learning, namely Q-learning, to learn both these overtaking behaviors. We tested our approach in several overtaking situations and compared the learned behaviors against one of the best NPC provided with TORCS. Our results suggest that, exploiting the proposed behavior-based architecture, Q-learning can effectively learn sophisticated behaviors and outperform programmed NPCs. In addition, we also show that the same approach can be successfully applied to adapt a previously learned behavior to a dynamically changing game situation.