An investigation of analysis techniques for software datasets

The goal of the study was to investigate the efficacy of different data analysis techniques for software data. We used simulation to create datasets with a known underlying model and with non-Normal characteristics that are frequently found in software datasets: skewness, unstable variance, and outliers and combinations of these characteristics. We investigated three main statistically based data analysis techniques: residual analysis; multivariate regression; classification and regression trees (CART). In addition to the standard "least squares" version of the technique, we also investigated robust and nonparametric versions of the techniques. We found that standard multivariate regression techniques were best if the data only exhibited skewness. However, under more extreme conditions such as severe heteroscedasticity, the nonparametric residual analysis technique performed best. We also found that even when the analysis technique did not accurately recreate the true underlying model, the faulty model could generate reasonably good predictions. The study indicates that simulation is very useful technique for evaluating different data analysis techniques.