Personalized Adaptive Virtual Object Placement in AR for Nonspeaking Autistic Users Using Behavioural Cloning

Nonspeaking autistic individuals (“nonspeakers”) represent about one third of the autistic population, yet most lack access to an effective alternative to speech. This lack of effective communication significantly limits their access to educational, social, and employment opportunities. Some nonspeakers have learned to spell words and sentences by pointing to letters on a physical letterboard held in their field of view by a trained human assistant. While effective, this method relies on the assistant for positioning the letterboard, limiting user autonomy and privacy. We report here a system we developed that uses Behavioral Cloning (BC) to automatically and adaptively position a virtual letterboard in Augmented Reality (AR). By observing finger, palm, head, and physical letterboard poses during real-life interactions between a nonspeaker and their assistant, we train a BC Machine Learning (ML) model that can adapt the placement of a virtual letterboard for that user. Results from 11 experiments (3 emulated scenarios and 8 nonspeaking autistic participants) show that our approach can accurately replicate the actions of the human assistant of any given user, outperforming a non-ML baseline personalized placement policy in both positional and rotational accuracies. Further, our novel BC formulation overcomes traditional data-efficiency limitations, allowing us to achieve high accuracy with a modest training effort. This work represents a foundational step toward enabling more autonomous and private communication for nonspeakers.