Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation

Holistic 3D indoor scene understanding refers to jointly recovering the i) bounding boxes, ii) room layout, and iii) camera pose, all in 3D. The methods either are ineffective or only tackle the problem partially. this paper, we propose an end-to-end model that simultaneously solves all tasks in real-time given only a single RGB image. The essence of the method is to improve the prediction by i) parametrizing the targets (e. g. , 3D boxes) instead of directly estimating the targets, and ii) training across different modules in contrast to training these individually. Specifically, we parametrize the 3D object bounding boxes the predictions from several modules, i. e. , 3D camera pose and object. The proposed method provides two major advantages: i) The helps maintain the consistency between the 2D image and the 3D, thus largely reducing the prediction variances in 3D coordinates. ii) can be imposed on the parametrization to train different modules. We call these constraints "cooperative losses" as they enable joint training and inference. We employ three cooperative losses for 3D boxes, 2D projections, and physical constraints to estimate a consistent and physically plausible 3D scene. Experiments on the RGB-D dataset shows that the proposed method significantly outperforms approaches on 3D object detection, 3D layout estimation, 3D camera pose, and holistic scene understanding.