Assessing the effectiveness of compact language models in repairing code with motor accessibility issues

Background: With approximately one in six people experiencing a significant disability, the accessibility of software is of critical importance. However, developers lack the knowledge and skills needed to fill the gap in creating universally accessible software. Objective: Inspired by the recent success of large language models (LLMs) in code generation, we investigate if compact language models (CLMs) can detect and correct motor accessibility related issues in code to support developers in meeting accessibility guidelines within on-premises development environments. Methods: We compared eight off-the-shelf CLMs’ abilities to resolve motor-related accessibility errors in sample code. We used four prompt variations (two persona-based and two non-persona-based) for each code sample to determine how the prompt impacts the CLMs’ outputs. Three LLMs were used to evaluate and score these outputs. Results: We see that CLMs have the capability to detect and correct a number of the code errors we tested, with several models providing correct code with at least one prompt per metric 40% of the time. However such success appears inconsistent as models rarely pass with every prompt on a given metric. We also observe that the prompt variations had minimal impact on output. Conclusions: The CLMs may not yet be ready to reliably support developers with improving motor accessibility in their code, despite showing promise. We urge the community to work towards resolving the gaps so that these CLMs may become a more helpful, reliable part of the developers’ workflow.