Human performance estimating with analogy and regression models: an empirical validation

Most cost estimation models seem to be validated without testing human performance and using data sets from custom software projects where the software typically is sized in lines of code (SLOC) or function points. From a practitioner's point of view this research seems not to address some important aspects of IT projects that we observe: estimating in an industrial environment is performed by people, not models; COTS projects are increasing their market share replacing traditional custom software projects; and industrial projects use a large variety of metrics to size the project deliverables and estimate the costs. Estimation by analogy tools like ANGEL and multiple regression analysis provide the necessary flexibility in terms of choice of input parameters. We describe an experiment to evaluate human performance where the subjects were aided by analogy and regression tools respectively. 68 partners and managers in Andersen Consulting estimated 48 different COTS projects. The results in terms of MMRE indicate that users benefit from both tools, however more from regression models than from analogy models as ANGEL. Furthermore, the performance of the ANGEL tool itself is not superior to the performance of the regression model. This result is contradictory to previous studies that claim that ANGEL outperforms multiple regression.