Speak2UI: Voice-Driven Low-Level Mobile Interaction Using Context-Aware LLM Command Parsing

Touch-based mobile interfaces pose significant accessibility challenges in hands-free scenarios. Face and gaze tracking lack precision for small UI elements. Deep learning-based voice systems depend heavily on training datasets. Commercial voice assistants are restricted to high-level commands. To address this gap, we present Speak2UI, an LLM-based voice control system for hands-free mobile interaction. Speak2UI leverages pre-trained language models for context-aware command interpretation without application-specific training and enables direct voice control of low-level UI elements. We evaluate Speak2UI through two studies. First, we compare its command recognition accuracy and inference latency against Voicify using identical datasets and metrics. Second, we assess small UI element manipulation against touch interaction through controlled user studies in real-world mobile applications. Results demonstrate the potential of LLM-based voice-driven UI manipulation for hands-free mobile interaction. Further evaluation with users from the target population is needed to assess the accessibility benefits of the proposed approach.