Beyond the Data: Architectural Choices for Imbalanced Learning in Vulnerability Detection

Static code analyzers, such as deep learning-based techniques, have proven to be essential in the security testing process. These methods rely on training an effective and reliable deep learning model to detect weaknesses and vulnerabilities. In the domain of source code analysis, the use of an adequate dataset of vulnerable and non-vulnerable code is crucial. However, it is recognized that these datasets are difficult to obtain, contain noise such as duplications, and are often strongly imbalanced. We aim to investigate the impact of these datasets on performance across various models and settings. Our methodology involves implementing two editing strategies, which include both model-level adjustments via a redefined loss function and enhancements through dataset resembling-based balancing approaches. We analyze various learning-based models, including two transformer-based models, SVDet and LineVul, two graph-based models, IVDetect and JLineVD, and two large language models, LLama and Mistral. The experimental analysis shows that resampling strategies combining over-sampling and under-sampling can improve model performance by up to $$22%$$ and yield gains of up to $$1000%$$ in terms of MCC, a suited metric for imbalanced data evaluation. Appropriate resampling methodologies, loss-function refinements, and retrieval-augmented generation techniques improve the performance of graph-based models, sequence-based models, and large language models, respectively, for vulnerability detection. However, despite these improvements, comparative analysis across models reveals that resampling and loss-function adjustments affect model stability and are not always adequate to fully address the challenges posed by severely imbalanced datasets.