Artificial Intelligence (AI) systems increasingly support human development in domains such as coaching, education, and healthcare. Yet most remain disembodied, relying solely on text or speech while neglecting non-verbal cues that are central to human communication. This research advances the science of embodied AI by developing computational models that integrate facial expressions and biometric signals (heart rate, HRV, temperature, electrodermal activity) for robust, real-time affect recognition. Building on embodied cognition, polyvagal theory, multimodal machine learning, and affective computing, the study compares early, late, and hybrid fusion architectures for synchronizing heterogeneous data streams. A mixed evaluation design combines benchmarking against multimodal datasets with experimental trials in adaptive coaching contexts. The expected contribution is twofold: (1) scientific, novel multimodal fusion architectures and comparative insights into fusion trade-offs; and (2) applied, an embodied AI coaching prototype and ethical guidelines for biometric data use. This work bridges gaps in affective computing and paves the way for emotionally intelligent, context-aware AI systems.
Thabo Mosala (Wed,) studied this question.