The Multi-Modal Assistive Communication System (MMACS) is an open-source, browser-based artificial intelligence platform designed to eliminate communication barriers for individuals with speech, hearing, or motor impairments. Unlike existing assistive technologies that address only a single communication modality, MMACS unifies three distinct AI-powered communication channels — American Sign Language (ASL) gesture recognition, viseme-based lip reading, and interactive Morse code translation — within a single zero-installation web application that runs entirely on the user's device. All machine learning inference is performed client-side using MediaPipe and TensorFlow.js, ensuring that no biometric data, camera frames, or personal communication records are transmitted to any external server. This privacy-first architecture achieves compliance with GDPR and HIPAA frameworks by design, making MMACS suitable for deployment in healthcare, rehabilitation, and educational settings without data governance concerns. The ASL recognition module employs a MediaPipe Hands two-stage pipeline to extract 21 three-dimensional hand landmarks per frame at 30 FPS, classifying 30 discrete signs using a multi-layer perceptron with softmax output. A smart gesture state management algorithm prevents repetitive voice output through a 3-second cooldown mechanism, producing natural and fluent speech synthesis via the Web Speech API. The lip reading module tracks 468 facial landmarks through MediaPipe Face Mesh, extracting a 10-dimensional lip geometry feature vector including mouth aspect ratio (MAR) and processes it through a bidirectional LSTM with CTC decoding for word reconstruction. The Morse code module implements the ITU-R M.1677-1 international standard with keyboard, mouse-click, and touchscreen input, complete with Web Audio API beep synthesis and real-time dot-dash visualization. An interactive analytics dashboard built with Recharts displays usage statistics, translation history, and modality activity trends, all stored locally in the browser's localStorage with no external database dependency. Empirical evaluation with 50 diverse participants — including individuals with hearing, speech, and motor impairments — demonstrated ASL recognition accuracy of 94.2% (±2.1%), lip reading accuracy of 87.8% (±3.4%), Morse code accuracy of 99.5% (±0.3%), and end-to-end latency below 200 ms across all modalities. A System Usability Scale score of 81.4 (Excellent range) and 92% user satisfaction rate validated the platform's real-world utility. Built on React 18.3.1 with TypeScript, Vite 5.4.21, Tailwind CSS, and Shadcn/ui, the application is fully responsive across desktop, tablet, and mobile form factors and is compatible with Chrome 90+, Firefox 88+, Safari 14+, and Edge 90+. The production bundle is optimized to 656 KB JavaScript and 65 KB CSS, making it accessible on mid-range mobile devices over standard 4G connections. Deployment requires no server infrastructure — the application can be hosted instantly on GitHub Pages, Vercel, or Netlify. This release includes the complete source code, deployment documentation, and research paper submitted to IEEE Access, establishing MMACS as a reproducible open-source benchmark for privacy-preserving, multi-modal assistive communication technology.
V et al. (Tue,) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: