Explainable AI for Software Reliability: A Three-Dataset Benchmark of Defect, Incident-Severity and Change-Failure Prediction

In this study, three software reliability tasks are considered: predicting defect prone modules, software incidents severity classification, and predicting deployment change failures, and explainable artificial intelligence (XAI) is studied for those tasks. For four explicitly synthetic tabular datasets with 2,400 data points each, that is, 7,200 observations, four classifiers were tested: logistic regression, random forest, extremely randomized trees and histogram gradient boosting. We were able to generate 12 model task evaluations, with 600 instances left out for each task. The accuracy and the macro-F1 scores of logistic regression were 77.17% and 75.40% for defect prediction, 72.00% and 72.33% for incident severity classification, and 64.33% and 63.69% for change-failure prediction, respectively. There are three areas of examination in the evaluation: predictive performance analysis, error patterns analysis, and model interpretation analysis. The confusion matrix, the receiver operating characteristic area under the curve (ROC-AUC) and the balanced accuracy are all used to explore these areas, along with the Matthews correlation coefficient and the feature-importance analysis. Twenty publications on the topic are critically examined and used to put the benchmark into the context and its methodological choices. The study offers an evaluation protocol that conveys the explanation in terms that the reader can follow and a comparative simulation framework of three software engineering decision points, along with a discussion of the constraints of the shown tasks when it comes to deployment. It does not develop a new learning algorithm but rather it is methodological and comparative. The results reported here are only applicable to the data sets generated and external validity has not been established. Conclusions regarding the practical effectiveness and suitability to deployment of the operational software data should be based on an independent evaluation prior to their use.