Three and a half stages of crowd wisdom

Background: According to the wisdom of the crowd idea, crowds can be smart even when most of their members are not. Much of the literature assumes that wisdom is to be extracted from a crowd by averaging the numerical predictions or probabilistic estimates of its members. Purpose: We compare a broad range of aggregation procedures, from simple averaging methods to optimized dynamic models and neural networks. Research Design: We apply multiple aggregation methods — including simple averaging, optimized weighted averages, a dynamic Hegselmann–Krause model of social learning, and two neural network architectures — to a dataset of probabilistic judgments, evaluating performance by both classification accuracy and Brier scores. Study Sample: 376 individuals providing probabilistic judgments about 1,200 statements with known ground truth. Data Collection and/or Analysis: Performance of all aggregation methods was assessed using classification accuracy and Brier scores, with neural network architectures trained to learn an optimal aggregation function directly from the data. Results: More sophisticated aggregators yield better performance. While some simple averaging methods exceed the performance of the average participant, optimized weighted averages and the Hegselmann–Krause model achieve significantly higher performance. Two neural network architectures outperform all other methods by a large margin, reaching a level of accuracy vastly superior to that of even the best individual in the crowd. Conclusions: Crowd wisdom is best not thought of as a fixed property but rather as something that can be achieved to different degrees, depending on the method used for aggregating opinions.