On the Reliability of Indicators of Learning Curve Model Accuracy

Cost analysts, operations analysts, purchasing managers, among others, frequently use learning curve (cost progress curve, experience curve) models to estimate future costs. Typically models are fit to available data, model parameters estimated and then forecasts are made based on the fitted models. Typically measures of goodness of fit such as r2, F or standard error are seen as indicators of how well a model may be expected to forecast future costs. This paper investigates whether such “ex ante” indicators do describe actual model forecast accuracy. Are there conditions when such indicators can be trusted to reliably inform analysts concerning the predictive quality of the model? Arc there conditions when the same indicators may be misleading? Should analysts pay attention to certain indicators in some situations and different ones in other situations? Can indicators of model quality be used to choose among competing models? These questions are addressed by simulating cost data under varying conditions, estimating learning curve models from the data, and then comparing the ex ante indicators of model fit with actual after-the-fact measures of model forecast accuracy. The degree to which ex ante indicators appropriately signal actual model accuracy under varying conditions is assessed. Broadly, the findings indicate that: A measure of prediction interval is the most useful signal of the accuracy of a model. However, the prediction interval is imprecise, biased, and potentially misleading, depending in part on the conditions under which it is used. Additionally, while useful for screening out obviously poor models, indicators of model quality have little ability to identify which of multiple models may be most accurate. Attention needs to be paid instead to the circumstances in which a model may be used.