What type of study is this?

September 10, 2025

Prompt engineering study: comparing pre-service engineers to large language models in requirements generation

SEShanae EdwardsThe University of Texas at Dallas KNKenny Nonso-Anyakwo JSJoshua D. SummersBoston University

Key Points

Pre-service engineers outperformed large language models in generating varied and complete requirements.
The study evaluated the work of 66 pre-service engineers against 4 LLMs across multiple criteria.
Natural language requirements can often lead to ambiguity, making effective generation crucial.
Future efforts may include developing true human personas for further comparison with LLM outputs.

Abstract

ABSTRACT: The objective of this research is to compare the requirements generated by human participants and large language models (LLMs). Requirements are statements that capture the needs and desires from stakeholders and organize them into design parameters. These statements are expressed in natural language which may lead to incompleteness and ambiguity. Due to the recent advancements in the natural language model such as ChatGPT and Gemini as a tool for requirement generation, this study investigates the quantity, variety and completeness of requirements generated by 66 pre-service engineers and 4 LLMs. This is because in some design projects, stakeholder access may be limited. The results show that pre-service engineers outperformed LLMs in variety, quantity and completeness. Future work could involve developing and comparing true human personas to LLMs.

AIに質問

Bookmark

View Full Paper