Graph-based multimodal affect recognition in children using prototypical networks

Introduction Although physiological signals such as heart rate, perspiration, and facial muscle activity are recognized as markers of emotional events, precisely classifying affective states from these data remains a significant challenge. Addressing this issue is fundamental for developing advanced human-computer interaction and assistive technologies. While emotion recognition in adults has been extensively studied, it is less understood in children, necessitating focused research. Methods This study introduces a multimodal framework tailored for the emotion recognition of children. We used prototypical networks to learn discriminative embeddings from each physiological modality. These embeddings were then used to construct an adaptive k-nearest-neighbors (KNN) graph that models the interrelationships among affective conditions across the modalities. A graph neural network (GNN) leverages this structural representation for the final classification, improving performance by capturing the intrinsic relational context. Results Our proposed framework improved classification performance by 8%–10% compared to single-modality baselines and existing fusion approaches, achieving an overall accuracy of 83%. Discussion These results show that multimodal fusion and graph-based learning can accurately capture the complex interplay of biological signals in children, providing a more accurate approach to pediatric affective computing.