This is not primarily a methods paper, but rather a paper about the limitations of methods. Using a large data set of expert forecasts with realizations, it emerges that expert point forecasts vary widely, and the distribution of point forecast errors appears to be very fat tailed. This paper uses simulated data to show that when samples from fat-tailed distributions are averaged, the results may not converge: extreme “outliers” arise with a frequency sufficient to disrupt any trends toward (apparent) convergence. It also shows that real-world expert judgment data from a number of different domains, collected over a period of decades, exhibit such fat-tailed behavior as regards point forecasts. Methods based on the “wisdom of crowds” that aggregate point forecasts of multiple experts are unlikely to yield good performance in such cases, regardless of the precise method by which judgments are weighted and aggregated. Moreover, increasing the number of experts used in such studies does not improve performance but rather degrades it, as does increasing the “diversity” of expert panels. However, when experts provide probabilistic forecasts, both the number of experts and diversity are helpful for the panel’s overall statistical accuracy, which in turn correlates positively with lower forecast error. The moral of the story is that point estimates elicited from multiple experts should not be aggregated. Decision analysts should instead rely on aggregating probability distributions elicited from experts rather than point estimates, as is already considered best practice. Weighting based on the experts’ measured performance yields even better results.
Roger M. Cooke (Fri,) studied this question.