GazeLink: A multi-language low-cost mobile eye-gesture communication system with large language models for people with amyotrophic lateral sclerosis

Amyotrophic Lateral Sclerosis (ALS) patients who have severe motor and speech impairments mostly rely on their eyes and assistive technology to communicate. However, existing high-tech products are expensive and hard to access, while low-tech products are inefficient and restrictive. To mitigate the limitations, this research proposes GazeLink, a multi-language low-cost mobile application for ALS patients to communicate efficiently with only eye movements. First, the system recognizes user eye gestures like left or up with machine learning and a template-matching algorithm. Then, it converts the eye gestures to words through a keyboard that supports English, Spanish, and Chinese. For efficiency, the system employs Large Language Models (LLMs) to generate a suitable sentence with words typed by the user and the context. Finally, the system provides text-to-speech and social media post services for both verbal and digital eye-gesture communication. Simulations conclude that sentence generation with LLMs can reduce user keystrokes by 81% while maintaining 90% of semantic similarity. Usability studies with 30 participants show that GazeLink can recognize eye gestures with 94.1% accuracy in varying lighting. After rapidly learning the user interface in under 10 attempts, first-time participants typed sentences of various lengths with their eyes at 15.1 words per minute, which is 7.2x faster than the common low-tech solution E-Tran. Experiments demonstrate GazeLinks efficiency, learnability, and accuracy in eye-gesture text entry. The system is extremely affordable (less than $0.1 a month), portable, and easily accessible online. It also supports different users, lighting, smartphones, and languages. Product testing with ALS patients and personalized LLM models will be the next step.