Evaluating a Large Language Model on Searching for GUI Layouts

The field of generative artificial intelligence has seen significant advancements in recent years with the advent of large language models, which have shown impressive results in software engineering tasks but not yet in engineering user interfaces. Thus, we raise a specific research question: would an LLM-based system be able to search for relevant GUI layouts? To address this question, we conducted a controlled study evaluating how Instigator, an LLM-based system for searching GUI layouts of web pages by generative pre-trained training, would return GUI layouts that are relevant to a given instruction and what would be the user experience of (N =34) practitioners interacting with Instigator. Our results identify a very high similarity and a moderate correlation between the rankings of the GUI layouts generated by Instigator and the rankings of the practitioners with respect to their relevance to a given design instruction. We highlight the results obtained through thirteen UEQ+ scales that characterize the user experience of the practitioner with Instigator, which we use to discuss perspectives for improving such future tools.