Cross-Dataset Evaluation of Imbalance Handling Strategies for Software Defect Prediction Using Machine Learning

Software defect prediction plays a critical role in enabling risk-aware software testing and improving overall software quality. However, real-world defect datasets are highly imbalanced, with defective modules typically representing a small minority of the total data. This imbalance significantly affects the performance of machine learning models, particularly in detecting defect-prone modules. In this study, we investigate the impact of imbalance handling strategies on defect prediction performance across multiple publicly available NASA datasets (CM1, KC1, PC1, and JM1). Three strategies are compared: no imbalance handling, class-weighted learning, and Synthetic Minority Over-sampling Technique (SMOTE). Using Logistic Regression as the baseline classifier and employing stratified 5-fold cross-validation, we evaluate performance using Area Under the ROC Curve (AUC) and recall for defective modules. Experimental results demonstrate that class-weight balancing consistently achieves the highest recall (up to 0.72) across datasets while maintaining competitive AUC performance. Statistical testing confirms significant differences between imbalance strategies. The findings highlight the importance of imbalance-aware learning in practical defect prediction systems and provide empirical guidance for integrating machine learning into intelligent, risk-based software testing frameworks.