Moment alignment transformer for video-to-video moment retrieval

Video-to-video moment retrieval ( Vid2VidMR ) is the task of localizing unseen events or moments in a target video using a query video. This task is challenging due to the need for semantic frame-level alignment and modeling complex dependencies between query and target videos. To address this, we introduce MATR (Moment Alignment TRansformer), a transformer-based model that captures semantic context and temporal detail for precise moment localization. MATR conditions target video representations on query features through dual-stage sequence alignment, encoding the necessary correlations and dependencies. These representations guide foreground/background classification and boundary prediction heads to identify moments in the target video that semantically match the query. To provide strong task-specific initialization, we further propose a self-supervised pre-training objective where the model learns to localize random clips within videos. Extensive experiments show that MATR achieves absolute gains of 15.4% in R@1 and 11.4% in mIoU over state-of-the-art methods on the ActivityNet-VRL dataset, and 16.4% in R@1 and 16.3% in mIoU on our newly proposed SportsMoments dataset. Code is available at: https://github.com/vl2g/MATR .