The Dynamic Optimization of AFL Fuzzing Mutation Strategies Based on Adaptive Reward Proximal Policy Optimization

Fuzzing has emerged as a vital technique for discovering vulnerabilities in software systems. American Fuzzy Lop (AFL) is a widely used fuzzer known for its effectiveness and efficiency. However, optimizing fuzzing strategies to enhance coverage and bug discovery remains a significant challenge. This paper introduces an innovative approach by integrating an Adaptive Reward Mechanism within the Proximal Policy Optimization (PPO) framework to improve AFL fuzzing performance. Our method dynamically adjusts rewards based on the fuzzing feedback, guiding the learning agent to prioritize paths with higher potential for uncovering vulnerabilities. Experimental evaluations demonstrate a clear trend of improvement in both code coverage and unique crashes found as model iterations increase. The Adaptive PPO method consistently outperforms the traditional AFL and standard PPO techniques across all iterations. By the 100th iteration, Adaptive PPO achieves 90% code coverage and discovers 25 unique crashes, compared to AFL's 60% coverage and 11 crashes, and PPO's 78% coverage and 17 crashes. This data highlights the effectiveness of the adaptive reward mechanism in enhancing exploration and vulnerability detection capabilities, supporting the assertion that Adaptive PPO is a superior approach for fuzz testing. The incremental improvements observed with each iteration underscore the method's potential for sustained performance gains over time, advancing the state-of-the-art in software testing and vulnerability analysis. Additionally, performance overhead analysis shows the method’s practical feasibility." to emphasize overhead validation.