Probabilistic Methods for Evaluating Human and LLMs During Design Problem-Solving

We present a probabilistic method for assessing design reasoning in design problem settings using soundness and completeness as metrics. Building on how inference mechanisms are employed during latent need elicitation from product reviews, we compare human-led and Large Language Models (LLMs) via protocols, workshops, and surveys. We demonstrate that human reasoning patterns tend to leverage user opinions, achieving deeper coverage of need potential, whereas LLMs often produce narrower, categorically constrained needs. These findings highlight the importance of balancing inference mechanisms to ensure both coherent reasoning steps and comprehensive exploration of the design space. By formally framing reasoning during design problem-solving, we offer a foundation for developing design enabled AI and deepens our understanding of how complex reasoning unfolds in practice.