Cognitive Assistance for Prosopagnosia Patients Using Landmark-Based Identity and Emotion Recognition

Prosopagnosia is a neurological condition that significantly affects the social interaction and quality of life of individuals. Existing assistive systems mainly focus on either face identity recognition or face emotion recognition, limiting their effectiveness in cognitive assistive scenarios. To overcome these, an integrated framework is needed that jointly addresses identity and emotion recognition to efficiently support prosopagnosia patients, as discussed in this study. The proposed system includes two separate modules: a face identity recognition module and a face emotion recognition module. The proposed system detects and aligns faces using Multi-Task Cascaded Convolutional Networks (MTCNN) with five-point landmark alignment. Face identity recognition is performed using an EfficientNet-B3 backbone to extract 512-dimensional facial embeddings, which are matched against a Structured Query Language (SQL) database using cosine similarity. Then, facial landmarks are detected along with emotion recognition, using the Dlib library, and are structured as a graph for High-order Graph Attention Network (HoGAN)-based relational interactions detection. The proposed system is trained using a joint loss function to efficiently provide real-time assistive feedback. The system achieves high recognition performance, with AUC values of 99.8% and 99.5% in both face identity recognition and face emotion recognition modules.