Keyframe selection from motion capture data with dual-agent reinforcement learning

Animation production workflows centred around motion capture techniques require animators to edit motions based on a set of keyframes. However, most existing keyframe selection methods are optimisation-based, which suffer from the issues of flexibility and efficiency. In this paper, a novel deep reinforcement learning method with dual agents are proposed for unsupervised keyframe selection. First, an S-Agent and an R-Agent evaluate the actions of selection and refinement, respectively. A deep spatio-temporal network, namely graph keyframe evaluation network (GKEN), is proposed for the agents. Then, an animation specified reward is devised based on reconstruction, which fulfills three important properties of the animation workflow: incremental reward, order insensitivity and non-diminishing returns. During the inference, it is no longer necessary to compute the reconstruction, which significantly decreases the run-time latency. Experiments on the CMU MoCap dataset demonstrate the efficiency of the proposed method without clearly compromising the effectiveness compared with the state-of-the-art methods. • A deep reinforcement learning with dual-agent to identify motion keyframes. • A spatio-temporal deep agent with graph convolutions and transformers. • Comprehensive experiments and human demonstrations for MoCap keyframing.