What question did this study set out to answer?

The aim is to develop a hands-free interaction system that utilizes gesture and voice control for desktop applications.

February 8, 2026Open Access

Octargon V11.0: A Multimodal AI-Powered Gesture and Voice Controlled Desktop Interaction System

Puntos clave

The aim is to develop a hands-free interaction system that utilizes gesture and voice control for desktop applications.
Developed a multimodal system integrating gesture recognition and voice commands.
Used MediaPipe for hand landmark tracking and Intel RealSense for depth filtering.
Implemented an AI assistant, Bee AI, for processing voice instructions.
Established a distributed document transfer system using Raspberry Pi for file sharing.
Designed the interface with PyQt5 for real-time interaction feedback.
Reliable detection of gestures like swipe and pinch for navigation and control.
Users can open applications and navigate documents via voice commands.
System allows wireless document sharing between computers on the same network.
Feedback mechanisms ensure users receive real-time updates on system status.

Resumen

Octargon V11.0 is a multimodal desktop interaction system designed to enable hands-free control of presentations, documents, and applications through gesture recognition, voice interaction, and distributed document transfer. The system integrates perception modules, artificial intelligence processing, and network communication into a unified desktop environment that allows users to interact with digital content in an intuitive and natural way. The system operates by combining visual sensing, audio processing, and intelligent command execution. Hand gestures are detected using computer vision techniques based on MediaPipe hand landmark tracking, supported by depth filtering from the Intel RealSense D455 camera. This enables reliable detection of gestures such as swipe navigation, pinch-based zoom control, mouse movement, and gesture-triggered document transfer commands. Voice interaction is provided through an AI assistant named Bee AI, which processes spoken instructions using a speech pipeline that includes audio capture, voice activity detection, Whisper-based speech transcription, and AI command interpretation. This allows users to control system functions through natural language commands such as opening applications, navigating documents, adjusting system volume, and initiating document transfer operations. The system also incorporates a distributed document transfer mechanism that enables wireless sharing of files between computers on the same network. A Raspberry Pi device functions as an intermediate relay server that coordinates communication between sender and receiver systems, ensuring reliable document exchange and simplified device discovery. All interaction modules are integrated into a desktop interface developed using PyQt5, which provides real-time feedback on gesture detection, voice interaction status, transfer progress, and system activity logs. By combining gesture interaction, voice-based AI control, and distributed communication capabilities, Octargon V11.0 provides a unified intelligent interaction platform suitable for presentation environments, educational settings, and digital exhibition systems.

Preguntar a la IA

Me gusta

Guardar

Ver artículo completo