What question did this study set out to answer?

The aim is to develop a framework that effectively integrates audio, visual, and semantic cues for coherent video summarization.

June 10, 2026Computer Modeling in Engineering & Sciences0 citationsOpen Access

Causal Cross-Modal Context Fusion for Real-Time Video Summarization with Predictive Tracking and Validated Adaptive Evaluation

ASAravapalli Rama SatishSRM University SVSai Babu VeesamSRM University SBShonak BansalChandigarh University

Key Points

The aim is to develop a framework that effectively integrates audio, visual, and semantic cues for coherent video summarization.
Developed a Cross-Modal Context Fusion framework with five integrated components.
Implemented a Temporal-Causal Graph Memory Network for capturing causal relations in multimedia.
Conducted experiments on SumMe, TVSum, and long-form documentaries to evaluate the system.
Achieved up to 15% F1 score and 12% ROUGE-L score improvements in summarization quality.
Human evaluations indicated 18% higher perceived coherence compared to previous methods.
Maintained >90% sustained approval rating in execution.

Abstract

Real-time video streams now flood everything from security cameras to social media, yet current summarization systems still stumble when audio, visual, and semantic cues unfold with tangled cause–and–effect patterns. Most cross-modal transformers treat correlations as if time were a flat canvas, ignoring how an early sound might trigger a later visual event in the process. They also lack mechanisms to predict tracking uncertainty, adapt to narrative shifts, or evolve their own evaluation criteria, leaving summaries brittle and often incoherent in process. To address these gaps, we propose a Cross-Modal Context Fusion framework built from five tightly linked components. A Temporal-Causal Graph Memory Network captures directional cause–and–effect edges across audio, video, and semantic signals, improving the logical flow of detected key segments. Additionally, a Predictive Entropy Reinforcement Engine learns camera focus and keyframe decisions that minimize future uncertainty, thereby stabilizing tracking under rapid motion or noise. The Cross-Modal Residual Synergy Transformer explicitly models discrepancies, such as off-screen speech, and feeds those residuals back to refine the fusion. For long-form narrative coherence, a Dynamic Hierarchical Context Predictor alternates between micro-actions and macro-story arcs, balancing fine detail with global structure. Finally, a Self-Evolving Evaluation Loop meta-learns to adjust loss weights as deployment contexts shift, sustaining performance without costly full retraining sets. Experiments on SumMe, TVSum, and long-form documentaries indicate up to 15% F1 and 12% ROUGE-L gains, with human studies reporting 18% higher perceived coherence and >90% sustained approval in process. The result is a video summarizer that reasons causally, anticipates uncertainty, adapts its own metrics, and delivers concise yet narratively faithful summaries suited for demanding real-time applications.

Ask AI

Helpful

Bookmark

View Full Paper