The Crowd Without People: Multi-agent Large Language Models as Idea Evaluators

Large Language Models (LLMs) are transforming the innovation process by scaling ideation and creativity. This yields the challenge of evaluating the growing flood of ideas. In simpler contexts, tasks with clear criteria and well-defined outcome possibilities, where human evaluators are also a bottleneck, LLM systems are already being used as a scalable and cheap alternative. In this study, we assess whether LLMs and LLM agents can be used as evaluators in innovation processes, making complex evaluations that require probabilistic judgments about future outcomes. Thereby, we examine the effect of different system, group, and collaboration architectures on the performance. Results indicate that implementing more LLMs instead of a single LLM is beneficial, traceable to wisdom of the crowd effects. Moreover, our results show that agent heterogeneity and structured collaboration provide added value, leveraging collective intelligence effects that further enhance evaluation performance beyond what stronger models achieve individually. Findings contribute to the theoretical understanding of LLM system architectures and discuss the practical relevance of LLM evaluators.