VulD-Transformer: Source Code Vulnerability Detection via Transformer

The detection of software vulnerability is an important and challenging problem. Existing studies have shown that deep learning-based approaches can significantly improve the performance of vulnerability detection due to their powerful capabilities of automatic learning semantically rich code representation. However, the deep learning-based source code vulnerability detection methods still have limited learning ability for remote contextual dependency information between code statements. In this paper, we propose a deep learning-based code slice-level vulnerability detection via Transformer, dubbed VulD-Transformer, which is designed to detect vulnerabilities more effectively. In VulD-Transformer, transformer model is used to capture the critical features of vulnerabilities of long code slices. Especially, we firstly obtain code slices containing data dependencies and control dependencies by extracting the vulnerability syntax features and programs’ Program Dependency Graphs. Moreover, in order to improve the feature learning capability of the model for remote code statements, we design a Transformer-based vulnerability detection model. The experimental results on four synthetic datasets show that, compared to the VulDeePecker, SySeVR-BGRU, SySeVR-ABGRU and Russell approaches, VulD-Transformer achieves 6.12%, 8.01%, and 7.63% improvement on average in accuracy, recall and F1-measure respectively, when the code slices are more than 256 tokens. In addition, compared with these baselines, VulD-Transformer achieves 9.01%, 38.51%, and 20.98% improvement on average in accuracy, recall and F1-measure respectively on two real source code vulnerability datasets, Devign and REVEAL respectively, which are significantly higher than those of the comparison methods.