A structured overview and empirical evaluation of fine-tuning pre-trained transformer models for software defect prediction

Software defect prediction (SDP) is an important task in software engineering that aims to identify defect-prone software components before deployment, thereby supporting software reliability and reducing maintenance effort. Recent advances in pre-trained transformer models (PTMs) have demonstrated considerable potential for source-code understanding and automated defect detection. However, comparisons of transformer-based models across previous studies are often complicated by differences in datasets, preprocessing procedures, training configurations, and evaluation protocols. This study presents a focused literature overview and a controlled empirical benchmarking evaluation of four widely used transformer-based models BERT Base, DistilBERT, RoBERTa, and CodeBERT—for software defect prediction using the CodeXGLUE-Devign dataset. Rather than proposing a new model architecture, the study investigates the comparative behavior of existing pre-trained transformer models under a common experimental configuration. To reduce dependence on a single random initialization, each model is independently fine-tuned and evaluated using five random seeds, resulting in 20 training runs. Performance is assessed using Accuracy, Precision, Recall, and F1-score, with results reported as mean ± standard deviation across the five runs. Pairwise differences are further examined using the Wilcoxon signed-rank test with Holm correction for multiple comparisons. The experimental results show that CodeBERT achieves the highest mean Accuracy (72.69 ± 0.88%), Precision (74.33 ± 1.75%), and F1-score (66.60 ± 1.67%), whereas BERT Base obtains the highest mean Recall (61.48 ± 3.07%). DistilBERT achieves an Accuracy of 69.29 ± 0.87% and an F1-score of 64.33 ± 1.48% while requiring substantially less training time than the larger models. RoBERTa exhibits lower and considerably more variable performance across the five runs. Nevertheless, none of the pairwise performance differences remain statistically significant at α = 0.05 after Holm correction. The findings demonstrate the importance of repeated-run evaluation, statistical analysis, class-wise assessment, error analysis, and computational considerations when benchmarking transformer-based models for SDP. The study therefore provides replication-oriented empirical evidence concerning the performance, stability, error characteristics, and computational trade-offs of existing transformer architectures under a controlled experimental setting.