Lightweight Multimodal Video Behavior Pattern Recognition Method Based on Language-Vision Contrastive Learning

Based on Contrastive Language-Image Pre-training (CLIP), this paper proposes a lightweight multimodal model for edge-oriented video behavior pattern recognition. The model extends CLIP from two coordinated aspects: low-cost temporal modeling in the visual encoder and video-adaptive prompt learning in the language branch, so as to learn generalizable multimodal video representations under constrained computing, memory and power budgets. Specifically, a lightweight Virtual-frame Interaction Module (VIM) is designed to obtain virtual frame tokens from the class tokens of sampled frames and then perform temporal convolution and virtual-frame token shift, enabling inter-frame interaction with limited additional parameters and FLOPs. A Global Frame Fusion Module (GBM) aggregates frame-level temporal representations into video-level representations, while a Visual-Reinforcement Prompt Module (VPM) fuses class-token and visual-token information through attention to enhance language semantics without substantially increasing model scale. Fully supervised experiments on four public video datasets, together with few-shot and zero-shot experiments, verify the effectiveness and generalization of the proposed method. Edge deployment experiments on NVIDIA Jetson Orin Nano further show that the proposed model achieves a favorable accuracy-latency-energy trade-off, reaching 85.7% Top-1 accuracy, 32.6 ms latency, 30.7 FPS and 224.94 mJ energy per video at T=8; the adaptive inference variant further reduces latency to 24.1 ms and energy consumption to 151.83 mJ with only a 0.4% accuracy drop.