A High-Acceptance-Rate VxWorks Fuzzing Framework Based on Protocol Feature Fusion and Memory Extraction

With the widespread application of Internet of Things (IoT) devices, the security of embedded systems faces severe challenges. As an embedded operating system widely used in critical mission scenarios, the security of the TCP stack in VxWorks directly affects system reliability. However, existing protocol fuzzing methods based on network communication struggle to adapt to the complex state machine and grammatical rules of the TCP. Additionally, the lack of a runtime feedback mechanism for closed-source VxWorks systems leads to low testing efficiency. This paper proposes the vxTcpFuzzer framework, which generates structured test cases by integrating the field features of the TCP. Innovatively, it uses the memory data changes of VxWorks network protocol processing tasks as a coverage metric and combines a dual anomaly detection mechanism (WDB detection and heartbeat detection) to achieve precise anomaly capture. We conducted experimental evaluations on three VxWorks system devices, where vxTcpFuzzer successfully triggered multiple potential vulnerabilities, verifying the framework’s effectiveness. Compared with three existing classic fuzzing schemes, vxTcpFuzzer demonstrates significant advantages in test case acceptance rates (44.94–54.92%) and test system abnormal rates (23.79–34.70%) across the three VxWorks devices. The study confirms that protocol feature fusion and memory feedback mechanisms can effectively enhance the depth and efficiency of protocol fuzzing for VxWorks systems. Furthermore, this approach offers a practical and effective solution for uncovering TCP vulnerabilities in black-box environments.