Voice and Hand Gesture-Based Multimodal Intelligent Assistant for Human-Computer Interaction

The traditional modality used for Human-Computer Interaction (HCI) involves keyboard and mouse input, which limits the naturalness and accessibility of human interactions with computing systems. This paper describes the design and implementation of a multimodal intelligent assistant that allows hands-free communication with computers through voice interaction, hand gesture recognition, and artificial intelligence. The system consists of speech recognition, a Large Language Model (LLM) reasoning component for providing answers to natural-language queries, a Text-to-Speech (TTS) component for spoken answers, a voice dictation component for hands-free typing, screen awareness component with AI-vision capabilities for analyzing screen content, and a lightweight floating UI for showing the state of the assistant. Hand gesture recognition is implemented as an additional way to perform control actions along with natural-language voice commands. Multilingual switching between English and Telugu languages is implemented explicitly to avoid unreliable language detection. This paper presents the architecture and methodology of the proposed system and current status of its implementation, specifying implemented components, components in development, and proposed for future development. Instead of inventing performance metrics, The paper highlights qualitative findings based on tests carried out on the functioning prototype. The objective is to show how the combination of voice, gestures, and AI-based reasoning can bring human computer interaction to a more natural and inclusive model.