Gesture as Code: Rethinking Programming Through Multimodal Interaction with AI for Apraxia Therapy

This position paper proposes a speculative re-conceptualization of programming as a dialogic, embodied, and adaptiveprocess shaped by gesture, speech error, and recovery. Drawing from the development of AURA, a research prototypedesigned to assist individuals with Apraxia of Speech, we explore how AI systems can redefine programming throughmultimodal interaction, adaptive feedback loops, and gesture-based inputs. Rather than focusing solely on codegeneration, we frame co-programming with AI as an interaction that includes diverse forms of expression. We arguethat by reframing programming as a space of adaptive, inclusive interaction, especially for neurodivergent users, webroaden its ethical and design possibilities. Context & Presentation: I developed this research for the Aarhus Decennial Conference 2025 workshop: "The End of Programming as We Know It - Envisioning Radical Re-Conceptualizations of Co-Coding with AI." This work contributes to the global discourse on alternative futures for AI-supported programming, focusing specifically on how generative-AI tools can evolve to support diverse human skills and neurodivergent experiences. Technical Innovation & Contribution: In this paper, I detail the conceptual and technical architecture of AURA, a research prototype I am developing to bridge the gap between High-Level Human-Computer Interaction (HCI) and Machine Learning. My work focuses on three primary innovations: Multimodal Data Integration: I design systems that process non-verbal, gesture-based, and speech-error inputs as first-class programming "code." Adaptive Feedback Loops: I am developing interaction models utilizing BiLSTM and CNN architectures that learn and adapt from user recovery patterns in Apraxia therapy. Inclusive Design Framework: I propose a "Gesture-as-Code" methodology that expands the definition of "programming" to include embodied, non-literate interaction modalities. Research Impact & Future Direction My work represents a critical shift toward Inclusive AI, positioning my research at the intersection of AI development and Special Educational Needs (SEN) therapy. By reframing programming as an adaptive, dialogic process, I aim to provide a scalable framework for future assistive technologies. I am focused on leveraging Reinforcement Learning and multimodal AI to empower neurodivergent communities and create more equitable digital futures.