Key points are not available for this paper at this time.
Applications of large language models (LLMs) in software engineering are soaring quickly. Despite ample evidence supporting LLMs’ capabilities of generating high-quality and creative suggestions, human-LLM interactions in software testing remain under-explored, and there lacks empirical evidence supporting efficacious human-LLM system designs. In software testing, the industry’s pursuit of “autonomous” systems neglects intensive human involvement in prior-design and post-review processes. This paper discusses two empirical user studies for a test case brainstorming task: (1) exploring user behaviors in human-LLM interactions compared to web search (\ (N₁=16\) ) and (2) investigating three modified interaction strategies—preemptive prompting, buffered response and guided input (\ (N₂=24\) ). We consolidate nine cross-disciplinary metrics to quantitatively evaluate the holistic performance of the human-LLM system, covering three perspectives: test quality, creativity, and attention span. Our findings reveal that users spend 126% more time interacting with LLMs compared to Google search. Moreover, preemptively prompting the LLM system significantly improves test quality and task creativity by over 30%, while simultaneously reducing user idle time by up to 49%. Based on the results, this paper discusses three interaction design principles—mixed initiative, acceptability, and appropriation—as guidance for future iterations of an efficacious LLM-assisted software testing system.
Shi et al. (Thu,) studied this question.