Scene Transformer: A unified architecture for predicting multiple agent trajectories

Predicting the motion of multiple agents is necessary for planning in dynamic. This task is challenging for autonomous driving since agents(e.g. vehicles and pedestrians) and their associated behaviors may be diverse influence one another. Most prior work have focused on predicting futures for each agent based on all past motion, and planning these independent predictions. However, planning against independent can make it challenging to represent the future interaction between different agents, leading to sub-optimal planning. In work, we formulate a model for predicting the behavior of all agents, producing consistent futures that account for interactions between. Inspired by recent language modeling approaches, we use a masking as the query to our model, enabling one to invoke a single model to agent behavior in many ways, such as potentially conditioned on the or full future trajectory of the autonomous vehicle or the behavior of agents in the environment. Our model architecture employs attention to features across road elements, agent interactions, and time steps. We our approach on autonomous driving datasets for both marginal and motion prediction, and achieve state of the art performance across two datasets. Through combining a scene-centric approach, agent permutation model, and a sequence masking strategy, we show that our model can a variety of motion prediction tasks from joint motion predictions to prediction.