An Automatic XSS Attack Vector Generation Method Based on the Improved Dueling DDQN Algorithm

As one of the most common web attack types, the XSS (Cross Site Scripting) attack is an important research topic in web attack and defense technology. However, the attack vectors in security evaluation methods are often based on expert experience or manual testing methods, which are not only costly and time-consuming but also have a large number of false positives. In this paper, we build an automatic XSS attack vector generation method based on the improved Dueling DDQN algorithm. First, we model the XSS attack vector generation process as a Markov decision process, mapping the initial attack vector mutation points and mutation strategies to the state space and action space of the model, respectively. Second, we propose an improved Dueling DDQN algorithm by introducing a priority experience replay mechanism to improve algorithm performance and the speed of attack vector generation. Third, we establish a feedback mechanism based on the edit distance algorithm to define the role of the reward function, preventing the model from getting stuck in local optima and achieving better mutation effects. Finally, we propose an automatic XSS attack verification method based on static semantic analysis to validate the effectiveness of our generated attack vectors. Based on the aforementioned methods, we have developed a prototype tool for automatic XSS scanning, which avoids generating a high proportion of invalid samples like traditional XSS scanners. The experimental results demonstrate that the improved Dueling DDQN algorithm outperforms other value-based reinforcement learning algorithms in terms of convergence speed, learning efficiency, and stability. The adaptive attack vector generation model can generate attack vectors that adapt to program context semantics and bypass defense mechanisms. Our method performs well when directly scanning the target system and exhibits a higher bypass rate of 85.71% in target systems deployed with WAFs. Furthermore, the model can learn the shortest path for selecting strategies to bypass WAF detection