A multimodal artificial intelligence system integrating facial expression voice emotion and behavioral movement analysis for assessing social-emotional competence in preschool children

Early identification of social-emotional difficulties in preschool children is critical for timely intervention, yet conventional assessment methods face limitations including subjective bias, static measurement, and resource intensity. This study developed and preliminarily evaluated a multimodal artificial intelligence system that integrates facial expression recognition, voice emotion analysis, and behavioral movement detection to assess social-emotional competence aligned with the Collaborative for Academic, Social, and Emotional Learning (CASEL) framework. The system employs attention-based fusion mechanisms to dynamically weight information streams and multi-task learning to jointly predict scores across five competency dimensions. A dataset comprising 500 typically developing children aged 36–72 months was collected from eight kindergartens, with expert annotations achieving inter-rater reliability above 0.85. On the held-out test partition, the proposed multimodal approach achieved 87.3% overall accuracy, outperforming single-modality baselines within the present sample. Correlation coefficients between system outputs and expert ratings ranged from 0.79 to 0.88, while test-retest reliability exceeded 0.84 across all dimensions. We interpret these results cautiously: the system was trained to reproduce expert consensus ratings rather than an independent clinical criterion; the evidence comes from a single regional sample of typically developing children; comparisons against state-of-the-art multimodal architectures were not conducted; and the held-out-institution check assessed environmental robustness rather than out-of-distribution generalization. The system is therefore positioned as a potential screening aid intended to support, not replace, professional judgment, with broader claims regarding objectivity, large-scale deployment, and clinical use remaining to be tested against independent cohorts and external outcomes in future work.