V-PLOT: Video-based Plot Type Classification and Captioning via Deformable-dilated Region Convolution Neural Network

Educational videos play a major component of the digital learning system that highlights the need for automated systems capable of understanding and indexing visual materials. However, existing methods cannot create a comprehensive table of contents for graphical objects within the videos and yet they remain difficult to detect and retrieve within lengthy lecture recordings. To address these issues, a novel V-PLOT approach is introduced to classify graphical plots with its summary from the keyframes of NPTEL educational videos. The proposed V-PLOT integrates Spatial Transformer Network (STN) to geometrically align the graphical regions present in the keyframes for improving downstream feature extraction. In the proposed Deformable-Dilated Region Convolution Neural Network (DD-RCNN), ResNeSt processes the aligned frames to extract the rich split-attention feature maps by capturing multi-scale structural features. Moreover, Deformable-Dilated Convolution (DDConv) enlarges the receptive field and sample’s irregular spatial locations to better capture the graphical plot details. Region Proposal Network (RPN) analyses the refined feature maps to generate bounding box proposals and RoIAlign ensures consistent dimensions of these proposals for plot classification with temporal timestamp mapping. Finally, the CLIP-based descriptive module assigns the most contextually relevant caption by comparing image embeddings extracted from text to generate clear descriptions. From the experimental validation, the proposed V-PLOT approach attains an accuracy of 98.65% using the collected NPTEL educational videos. Moreover, the proposed V-PLOT model increases the overall accuracy by 4.90%, 5.56%, 8.03% and 9.04% better than the OWL-ViT, QWEN-VL, HybridTabNet, and BiFPN respectively.