Crowdstorming You Can Trust: Assessing the Reliability of Crowdsourced Ideation and Prioritization

Asking crowds of stakeholders to generate and prioritize ideas has become commonplace, from citizen consultations to private open-innovation challenges. Although the reliability of the results of such consultations is taken for granted, their consistency across independent groups has never been systematically assessed (to our knowledge). Would running the same open-innovation challenge or citizen consultation with different groups of stakeholders allow decision-makers to extract the same best ideas? In this report, we rigorously assess the reliability of the results obtained with a professional crowdstorming method in a real-world setting. Two separate but equivalent groups of MBA students in the same course were asked to independently propose and prioritize ideas to improve the MBA program. Using semantic similarity measures based on OpenAI and Google's embeddings, we find that the two groups independently proposed similar sets of valuable ideas but different sets of poor ideas. We also find that they prioritized similar ideas similarly, enabling the decision maker to select similar sets of winning ideas in both groups. This suggests that a well-designed crowdstorming process may be trusted to produce results that accurately and reliably capture the collective intelligence.