WebRLED: A Deep Reinforcement Learning Approach for Automated Web Application Exploration

Automated web application exploration has been considered a challenging task, considering its large state space and complex interaction logic. Deep Reinforcement Learning (DRL) is a recent extension of Reinforcement Learning (RL), leveraging advantages of the powerful learning capabilities of neural networks, making it suitable for exploring a complex state space. In this paper, we propose WebRLED, an effective approach for automated web application exploration, leveraging the capabilities of deep reinforcement learning. WebRLED has the following characteristics: (1) a grid-based action value learning technique, which can improve the efficiency of state space exploration; (2) a novel action discriminator which can be trained during the exploration to identify more available actions; (3) an adaptive, curiosity-driven reward model, which considers the novelty of an encountered state within an episode and the global history, and can guide exploration continuously. We conduct a comprehensive evaluation of WebRLED on 12 open-source web applications. The experimental results show that WebRLED achieves higher code/state coverage compared to existing state-of-the-art (SOTA) techniques. In particular, on complex applications, WebRLED achieves relative improvements over the best baseline of 26.22% in code coverage, 82.59% in state coverage (WebEmbed), and 65.64% in state coverage (URL+structure). Additionally, WebRLED detects more failures on average than the best baseline, indicating that stronger exploration also improves failure exposure.