Large Language Models for Mobile GUI Text Input Generation: An Empirical Study

Mobile applications (apps) have become an essential part of our daily lives, making ensuring their quality an important activity. GUI testing, a quality assurance method, has frequently been used for mobile apps. When conducting GUI testing, it is important to generate effective text inputs for the text-input components. Some GUIs require these text inputs to move from one page to the next, which remains a challenge to achieving complete UI exploration. Recently, Large Language Models (LLMs) have shown excellent text-generation capabilities. Among the LLMs, OpenAI's GPT series has been widely discussed and used. However, it may not be possible to use these LLMs for GUI testing actual mobile apps, due to the security and privacy issues related to the production data. Therefore, it is necessary to explore the potential of different LLMs to guide text-input generation in mobile GUI testing. This paper reports on a large-scale empirical study that extensively investigates the effectiveness of nine state-of-the-art LLMs in Android text-input generation for UI pages. We collected 114 UI pages from 62 open-source Android apps and extracted contextual information from the UI pages to construct prompts for LLMs to generate text inputs. The experimental results show that some LLMs can generate relatively more effective and higher-quality text inputs, achieving a 50.58% to 66.67% page-pass-through rate, and even detecting some real bugs in open-source apps. Compared with the GPT-3.5 and GPT-4 LLMs, other LLMs reduce the page-pass-through rates by 17.97% to 84.79% and 21.93% to 85.53%, respectively. We also found that using more complete UI contextual information can increase the page-pass-through rates of LLMs for generating text inputs. In addition, we also describe six insights gained regarding the use of LLMs for Android testing: These insights will benefit the Android testing community.