ABSTRACT With the rapid globalization of Mandarin Chinese, the demand for effective oral training has shifted from static desktop applications to dynamic Real‐Time Communication (RTC) environments. Traditional systems often rely on unimodal acoustic analysis or high‐latency cloud‐based processing, which fail to provide the instantaneous feedback required for natural interaction. This paper proposes the Edge‐Assisted Multimodal Fusion Transformer (EAMFT), a novel framework specifically designed for the real‐time correction of Chinese pronunciation. Compared with representative Support Vector Machine (SVM)‐based approaches, EAMFT leverages a Cross‐Modal Attention (CMA) mechanism to synchronize acoustic features with visual articulatory movements (lip‐sync), significantly improving tonal classification accuracy in noisy environments. To address the strict latency requirements of RTC, we implement an edge‐cloud collaborative training policy and a Hysteresis‐based Correction Trigger (HCT) to ensure stable, nonintrusive feedback. Experimental results on the C‐Sino and L2‐ARCTIC corpora demonstrate that EAMFT achieves a recognition accuracy of 94.6% with an average end‐to‐end latency of only 65 ms, outperforming state‐of‐the‐art methods in both robustness and responsiveness.
Xiaofeng Jin (Mon,) studied this question.