Diverse Episodic Trajectory Exploration for Deep Q-Networks With Application to Self-Driving Cars

How to explore and when to exploit are indispensable considerations in the design of deep reinforcement learning (DRL) algorithms. This is of particular importance for ensuring successful convergence towards an optimal policy under the DRL framework. This paper presents a novel exploration strategy for Deep Q-Networks (DQN) aimed at enabling the agent to uncover high-reward regions within the state-space. The proposed strategy, termed Diverse Episodic Trajectory Exploration (DETE), introduces an exploration split point in each training episode, at which the agent is directed to explore alternative actions in addition to the one exploited by strictly following the policy dictated by the current Q-function. This diversifies the agent’s experienced trajectories, leading to more comprehensive policy learning. The values of the transitions encountered during the exploratory phase are subsequently estimated using a dedicated deep critic network, which regulates the loss function of the agent’s main network to improve overall policy optimization. The effectiveness of DETE is demonstrated by controlling a self-driving car navigating a multi-lane intersection scenario, which is simulated using the Carla platform.