Detection Method for Closed-Source VxWorks Memory Protection Mechanisms Based on Dynamic Instruction Translation Monitoring

In embedded real-time operating systems, memory protection mechanisms are critical for ensuring system security. However, for closed-source platforms like VxWorks, widely used in critical domains such as aerospace and industrial control, existing methods struggle to effectively detect the runtime status of memory protection mechanisms without access to source code. In contrast, research on memory protection mechanisms (e.g., ASLR and DEP) in Windows and Linux has developed into a mature field, highlighting the research intensity in this area. This paper proposes a detection method tailored for VxWorks, which instruments function call instructions at the QEMU TCG layer to dynamically reconstruct call chains and combines this with static modeling to automatically identify the activation status of key memory protection mechanisms, such as text segment write protection and stack non-executability. To validate the method’s effectiveness, three groups of firmware samples were designed, representing scenarios with no protection, partial protection, and full protection enabled. Experimental results demonstrate that the method delivers stable and reliable detection across various configurations, with no false positives or false negatives. Furthermore, open-source test cases enhance the credibility and reproducibility of the experiments. This approach, characterized by automation, non-intrusiveness, and high adaptability, provides an efficient tool for verifying the security configurations of closed-source embedded systems.