High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks

Predicting future video frames is extremely challenging, as there are many of variation that make up the dynamics of how frames change through. Previously proposed solutions require complex inductive biases inside architectures with highly specialized computation, including masks, optical flow, and foreground and background separation. In work, we question if such handcrafted architectures are necessary and propose a different approach: finding minimal inductive bias for video while maximizing network capacity. We investigate this question by the first large-scale empirical study and demonstrate-of-the-art performance by learning large models on three different: one for modeling object interactions, one for modeling human motion, one for modeling car driving.