Applications of large language models (LLMs) in software engineering are soaring quickly. Despite ample evidence supporting LLMs’ capabilities of generating high-quality and creative suggestions, human-LLM interactions in software testing remain under-explored, and there lacks empirical evidence supporting efficacious human-LLM system designs. In software testing, the industry’s pursuit of “autonomous” systems neglects intensive human involvement in prior-design and post-review processes. This paper discusses two empirical user studies for a test case brainstorming task: (1) exploring user behaviors in human-LLM interactions compared to web search (\ (N₁=16\) ) and (2) investigating three modified interaction strategies—preemptive prompting, buffered response and guided input (\ (N₂=24\) ). We consolidate nine cross-disciplinary metrics to quantitatively evaluate the holistic performance of the human-LLM system, covering three perspectives: test quality, creativity, and attention span. Our findings reveal that users spend 126% more time interacting with LLMs compared to Google search. Moreover, preemptively prompting the LLM system significantly improves test quality and task creativity by over 30%, while simultaneously reducing user idle time by up to 49%. Based on the results, this paper discusses three interaction design principles—mixed initiative, acceptability, and appropriation—as guidance for future iterations of an efficacious LLM-assisted software testing system.
Shi et al. (Fri,) studied this question.