Scene parsing using a prior world model

We present a new paradigm for constructing a 3D model of a scene from images. Our approach makes strong use of a prior 3D model of the scene. Changes from scene to scene are regarded as a Markov dynamical system, which is described by a probabilistic transition model. From the prior 3D scene model, the model of scene change dynamics, and a newly acquired image, we compute the new 3D scene model which is most consistent with the observed image and the changes from the prior model. The use of a prior 3D scene model allows the method to deal with complex scenes, maintain hidden state, respect object persistence, perform object segmentation, and provides computational efficiencies. In this paper we formalize a mathematical framework for physically consistent 3D scene models, and changes to scene models that preserve physical consistency. From this framework, we first derive a generic scene model optimization algorithm for the general 3D scene interpretation problem, and we then present a polynomial time approximation for this algorithm. We detail the implementation of the algorithm for range images computed by stereo imaging, and present extensive experimental results on sequences of scenes containing dozens of objects and multiple changes from scene to scene.