Key points are not available for this paper at this time.
Abstract Understanding the semantic meaning behind animal behavior remains a fundamental challenge in neuroscience and biomedical research. We present BehaVERT (BehaVior bERT), a transformer-based model that treats skeletal motion as structured language. By tokenizing movement into semantic units and learning temporal dependencies, BehaVERT decodes behavioral patterns from raw keypoint trajectories. Our pipeline addresses three critical gaps: accessibility, standardization, and interpretability. First, the pipeline provides web-based annotation tools for keypoint and behavior labeling that require no programming expertise, democratizing dataset creation. Second, this work contributes curated benchmarks with novel pose annotations and standardized formats, enabling reproducible evaluation across diverse behavioral tasks. Third, our transformer-based architecture processes temporal sequences to reveal latent meaning in movement, bridging the gap between immediate actions and long-term behavioral trends through both discrete behavior and continuous state classification. Novel training strategies including temporal unfolding and context-aware augmentation achieve state-of-the-art performance across five benchmarks: CalMS21, MABe22, PAIR-R24M, DeepEthogram, and SBeA. Beyond predictive accuracy, this work provides unprecedented interpretability: unsupervised clustering exposes learned behavioral structure, while attention analysis reveals which temporal moments and behaviors drive decisions. Our publicly accessible tools and analysis methods establish reproducible standards for decoding animal behavior.
Shin et al. (Mon,) studied this question.