Toward video-LLM driven workflow for behavioral segmentation and scoring in mice performing a skilled water-reaching task: an evaluation of recent LLM models

Manual behavior scoring is labor-intensive and subjective. Video-capable large language models (LLMs) offer a transformative, scalable solution for accelerating and standardizing neuroscience workflows. We benchmarked state-of-the-art video LLMs (Gemini 2.5 Pro, Qwen3-VL, and VideoLLaMA3) for automated behavioral segmentation and scoring of mice performing a water-reaching task. Videos of mice performing water reaching were analyzed by the LLMs. Accuracy was compared across different models and against prompt adjustments within Gemini. To assess classification determinants, video fidelity was altered through pixel interpolation and key regions blurred (paws/snout-mouth). In addition, the models were asked to describe the mouse’s actions over time. Finally, an open-source rat lever-pressing dataset was utilized to validate behavioral segmentation under a few-shot learning framework, assessing the impact of visual examples on the identification of discrete action sequences. Gemini 2.5 Pro (0.74±0.12 accuracy) and Qwen3-VL-30B (0.67±0.13) exhibited the ability to classify trial outcomes. Reliable classification required a minimum pixel resolution of 0.28 mm per pixel and careful consideration of the model frame tokenization rate. Accuracy is significantly reduced upon obscuring the snout-mouth area. In 549/1058 of videos, Gemini 2.5 Pro also provided completely accurate frame-to-frame behavior segmentations. The inclusion of visual examples improved model detection of user-defined behaviors. Video-LLMs offer potential to accelerate neuroscience by providing scalable, objective quantification of goal-directed behaviors. By producing temporal annotations, Gemini enables fast first-pass labeling that markedly streamlines manual dataset curation.