Large Language Models for Source Code Vulnerability Detection: A DiverseVul Analysis

Abstract Software's pervasive impact and increasing reliance in the era of digital transformation raises concerns about vulnerabilities. The emergence of code assistants for code generation further aggravates this issue, emphasizing the critical need for prioritizing software security. In recent research, DL has become a very promising instrument for vulnerability detection, with most recent approaches utilizing LLMs for Software Vulnerability Detection. The data utilized in this context is curated from real-world projects or synthetically, with the latter achieving great metric scores but poor performance in a real context. Driven by this issue, DiverseVul has been curated to be the largest dataset containing C/C++ vulnerable and non-vulnerable functions extracted from real-world projects. This work intends to explore this dataset by using it to fine-tune three LLMs (i.e.: CodeBERT, CodeGPT, and NatGen), evaluating their performance on vulnerability detection. During data processing, several erroneous data points were found, motivating the creation of a refined version of the dataset. Moreover, to establish a comparable baseline, the same models were fine-tuned on the RCVEFixes dataset, which is a refined version of the CVEFixes dataset containing only C/C++ functions. The results show that the best-performing models were CodeBERT trained on DiverseVul with 69% F1-Score and NatGen trained on RVCEFixes with 53%. It can be concluded that the performance of CodeBERT trained on DiverseVul is generally higher than average literature using similar techniques.