EBITH: A Smartphone-Based Multimodal Accessibility System for Blind and Deaf Users Using Computer Vision and Sign Language Recognition

Assistive apps usually serve one disability at a time, so a deaf person and a blind person have no common channel. We present EBITH, a smartphone system that connects deaf users of Pakistani Sign Language (PSL) with blind and hearing users in one conversation, and helps blind users move through indoor spaces in Pakistani institutions. It has three components. (1) A PSL sentence recogniser built on signer-normalised pose and hand landmarks: evaluated leave-one-signer-out on 15 everyday phrases, it reaches 96.7% top-1 accuracy (95% CI 93.3–99.2) and 100% top-3, and a confidence threshold calibrated on unseen signers answers 97.5% of clips with 99.1% precision. An appearance-based CNN–LSTM scores 82.7% on its original split but only 10.1% on unseen signers, which motivates the landmark design and signer-independent evaluation. (2) A 40-class indoor detector (YOLO11s) for classrooms, offices and corridors, including 19 navigation-relevant classes absent from COCO, such as doors, stairs and door handles. Non-exhaustive training labels made it ignore most chairs in real classrooms; completing them with a COCO teacher raises mAP50 on exhaustively labelled local photos from 0.250 to 0.441, and fine-tuning on a few dozen local photos raises it from 0.473 to 0.723 in 5-fold cross-validation. (3) A hazard-first scene description that speaks warnings such as stairs or a close obstacle first, with a distinct vibration; a photo is described in about 0.5 s. We will release the data-processing, training and evaluation code.