VideoGNN: Video Representation Learning via Dynamic Graph Modelling

Graphs offer a flexible structure for vision tasks, with CNNs and Transformers conditioned as two specific cases of graph structures. In CNNs, the input images are treated as graphs where only neighboring patches are connected, whereas Transformers view images as fully connected graphs. To leverage the potential of graphs in video representation learning, effective graph generation and training methods are crucial. To this end, we propose VideoGNN, which represents the video as a discrete time dynamic graph and learns the dynamic graph efficiently. Given the multitude of frames in videos, we introduce an efficient graph generation module characterized by low complexity and high quality, facilitating the transformation of videos into dynamic graphs. Additionally, we introduce a dual-view graph neural network to capture spatial and temporal information from the generated dynamic graphs. Then, a sequential model is applied to capture the long-term temporal information and generate the final frame embeddings. Experiments demonstrate that VideoGNN can achieve competitive results in terms of graph quality assessment and video downstream tasks. The codes are available at https://github.com/Dodo-D-Caster/VideoGNN .