An End-to-end Learning Framework for Visual Camera Relocalization Using RGB and RGB-D Images

Abstract Camera relocalization plays a vital role in the realms of machine perception, robotics, and augmented reality. Direct learning methods based on structures can have a learning-based approach that can learn scene coordinates and use them for camera position estimation. However, the two-stage learning of scene coordinate regression and camera position estimation can result in some of the scene coordinate regression knowledge being lost throughout the learning process of the final pose estimation system, thereby reducing the accuracy of the pose estimation. This paper introduces an innovative end-to-end learning framework tailored for visual camera relocalization by employing both RGB and RGB-D images. Distinguished by its integration of scene coordinate regression with pose estimation into a concurrent inner and outer loop during a singular training phase, this framework notably enhances pose estimation accuracy. Engineered for flexibility, it accommodates training with or without depth cues and necessitates merely a single RGB image during testing. Empirical evaluation substantiates the proposed method’s state-of-the-art precision, attaining an average pose accuracy of 0.019 m and 0.74° on the indoor 7Scenes dataset, together with 0.162 m and 0.30° on the outdoor Cambridge Landmarks dataset.