Contactless sensing for future hearing aid device

Hearing impairment affects approximately 5% of the global population, and this number is expected to rise to nearly 700 million by 2050. In the United Kingdom alone, around 11 million individuals experience hearing difficulties with age related hearing loss. Existing technologies; i.e., vision-based methods, and wearable sensors are designed to support non verbal communication. However, they have several limitations. Vision-based methods depend heavily on lighting and raise privacy concerns, and wearable sensors, such as those embedded in hearing aids, can be inconvenient, requiring regular maintenance and frequent charging. To overcome these challenges, this thesis explores Multimodal (MM) sensing approach for hearing impairments, focusing on sign language recognition, facial expression analysis, hand and head movement detection, phrase recognition in British Sign Language (BSL) and common signs in American Sign Language (ASL) and BSL. The research investigates the use of Radio Frequency (RF) sensing through radar and camera fusion to develop contactless, privacy-preserving system that is capable of interpreting non-verbal communication. Across three main studies, radar and video data were collected and analyzed using advanced Deep Learning (DL) model. The first study compared radar-only and video-only systems for fifteen signs that are common in ASL and BSL, where radar achieved 96% accuracy and video achieved 82%. The second study focused on recognising micro movements, i.e., head and hand movements, and facial expressions using radar sensing, achieving 94.2% accuracy. The final study used radar data, video data, and multimodal fusion to recognise phrases in BSL. Radar only achieved 95.47%, video-only achieved 89.72% and MM sensing achieved 93% classification accuracy. The proposed MM technique successfully demonstrate that RF sensing and multimodal fusion can accurately recognise non-verbal communication without physical contact or visual intrusion. This work establishes a proof of concept for next-generation, contactless MM hearing aid systems designed to promote independence, privacy, and accessibility for Deaf and hard-of-hearing individuals.