Quality versus quantity: including imperfect training data improves prediction of user satisfaction in software projects

This study explored the impact of data quality and quantity on predicting user satisfaction in software projects using the ISBSG dataset. This study evaluated the performance of machine learning models by varying the data variants, i.e. strategies for including low-quality data in the training subsets. A rigorous methodology was employed to ensure the reliability of the results and validate the conclusions. It involved repeated nested cross-validation, standardised accuracy, win–tie–loss analysis, matched-pairs rank biserial correlation, Scott–Knott effect size difference test, and Kendall’s W. The findings suggest that models predicting user satisfaction trained on larger datasets, despite containing low-quality data, usually outperformed those trained on smaller, high-quality datasets. Hence, expanding the training data can improve predictive accuracy, even if it comes at the cost of data quality. Support vector machines (SVM) and extreme gradient boosting trees delivered the most accurate predictions across all data variants. A combination of SVM with the data variant involving all cases and attributes, where at least 40% of values were valid (non-missing), was the most accurate. Obtained findings enabled the formulation of a preliminary recommendation: when working with ISBSG or similar datasets, researchers and practitioners prioritising predictive accuracy of user satisfaction should train models on relatively large datasets, even if they contain some low-quality data, especially when dealing with limited data availability. Provided rankings of data variants, models, and combinations may help set up future studies.