Answers are grounded in this paper and cite their sources.
The interactive film and television (FAT) system urgently requires real-time audience emotion understanding, but existing methods inadequately capture subtle emotional variations in dynamic audiovisual contexts. This study proposes a multimodal emotion recognition model for FAT scenarios, integrating facial expressions, vocal prosody, and audiovisual contextual features under real-time interaction constraints. It incorporates the Temporal Context Synchronization Attention (TCSA) mechanism to align user response sequences with audiovisual content features and capture stimulus–response causal associations, and the Dynamic Weighted Routing (DWR) strategy to adaptively adjust visual, auditory, and contextual fusion weights across FAT genres. On IEMOCAP and LIRIS-ACCEDE, the jointly trained model achieves 87.9% and 87.2% accuracy, with F1-scores of 87.8% and 87.0%. Under the same joint training setting, the overall accuracy and F1 score on the combined IEMOCAP+LIRIS-ACCEDE test set (approximately 2,310 samples) are 87.6% and 87.4%, respectively. These values represent the sample-size-weighted averages of the results from the IEMOCAP and LIRIS-ACCEDE test sets. Under the specific six-class relabeling scheme of IEMOCAP, recognition of Low-Arousal Sadness (LAS), a non-clinical state, improves from 62.3% with the two-stream baseline (without TCSA/DWR) to 74.2% with the jointly trained full DWR-TCSA model. This represents a 19.1% relative improvement over the two-stream baseline under the same relabeling scheme, calculated as (74.2%−62.3%)/62.3%. The 74.2% result is obtained by the jointly trained full DWR-TCSA model on the IEMOCAP test set, whereas LAS recognition further increases to 75.8% (i.e., 0.758) on the combined IEMOCAP+LIRIS-ACCEDE test set. Recognition of Surprise also improves from 82.3% to 84.9%. Across five random seeds, DWR-TCSA achieves an accuracy of 0.879 ± 0.004 (95% CI [0.874, 0.884]) and an F1 score of 0.878 ± 0.005 (95% CI [0.872, 0.884]). Under leave-one-session-out (LOSO) cross-validation, its advantages over the two-stream baseline are statistically significant. Five-fold paired t-tests (df = 4) yield t(4) = 25.16, p < 0.001, for accuracy and t(4) = 30.10, p < 0.001, for LAS recall. Both differences remain significant after Holm–Bonferroni correction for multiple comparisons. The model also achieves a higher mean accuracy across the five random seeds than the cross-modal attention baseline (0.879 ± 0.004 vs. 0.857 ± 0.009). After lightweight optimization, the model achieves an inference speed of 72 frames per second (FPS), with a latency of 186 ms. In a real-world experiment involving 96 participants, the system achieves a mean real-time accuracy of 86.2%. This accuracy refers to the agreement between the model’s window-level predictions and the reference labels. The reference labels are primarily derived from participants’ self-reports, supplemented by consensus judgments from blinded annotators. Therefore, they represent subjective rather than independently objective labels. The experiment also yields a mean interaction satisfaction score of 4.19 out of 5.00. In a separate online questionnaire survey involving 300 valid respondents, 91.7% (275/300) report being satisfied or very satisfied with the system’s emotion recognition accuracy. The 96 participants in the aforementioned real-world experiment were recruited from this pool of 300 respondents. The framework supports intelligent interactive media experiences.
No takes yet. Share an insight, caveat, or question.
Li et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: