Aligning Sound, Sight and Sense: Multimodal Semantic Understanding as a Driver of Cross-Modal Communication Effectiveness on Douyin

Short-video platforms have made tri-modal production—moving images, voice, and on-screen text—the default grammar of digital communication, yet much of this content suffers from a paradox we call modality surplus with semantic deficit: channels multiply while meanings drift apart. Drawing on multimodal discourse analysis and the cognitive theory of multimedia learning, this study conceptualizes multimodal semantic understanding (MSU) not merely as a recognition technique but as a measurable production resource, and asks whether semantic alignment across modalities improves communication outcomes. We constructed a Multimodal Semantic Consistency Index (MSCI) with a CLIP-style audio-visual-text alignment pipeline and applied it to a stratified sample of 600 Douyin videos across six content categories. Communication effectiveness was measured as a composite of completion rate, likes, comments, and shares. Hierarchical regression shows that semantic consistency is the strongest single predictor of effectiveness (β = 0.243, p < .001), raising explained variance by 23.8 percentage points over account- and format-level controls; the effect is significantly amplified in knowledge-intensive categories (β = 0.083, p < .001), and tri-modal production pays off only when consistency is high. The findings reframe platform-era communication as a problem of cross-modal meaning governance, and yield concrete strategies for creators, MCNs, and platform governance.