Multimodal collaborative modeling of teacher verbal and non-verbal classroom behaviors for teaching effectiveness prediction

Teacher classroom behavior sits at the centre of any serious attempt to understand and improve instructional quality, yet the verbal and the non-verbal channel are still typically examined apart, which leaves their pedagogically meaningful interdependence unmodelled. This paper proposes a multimodal deep learning framework that models the two channels collaboratively and predicts teaching effectiveness from their joint representation. Four modules compose the framework: temporal synchronization of the audio and video streams, parallel unimodal encoding through BERT-based language representation and SlowFast spatiotemporal networks, bidirectional cross-modal attention fusion regulated by a gated modality weighting mechanism, and a dual-head prediction network optimized jointly for effectiveness classification and continuous score regression. On a corpus of 186 classroom sessions recorded in 12 schools, the framework attains 82.3% classification accuracy, 80.1% macro-F1, an RMSE of 6.58 and a Pearson correlation of 0.879, ahead of ten comparison systems spanning unimodal, traditional machine learning and state-of-the-art multimodal fusion approaches. Ablation identifies bidirectional cross-modal attention as the component that matters most, and an attention analysis conducted across all test sessions indicates that verbal-gestural temporal synchrony accompanies higher observer-rated teaching effectiveness. The framework thus offers a scalable, automated route from multimodal teacher behavior to instructional outcomes, with potential applications in teacher professional development and intelligent classroom observation systems.