A System for Dense Monocular Mapping with a Fisheye Camera

We introduce a novel dense mapping system that uses a single monocular fisheye camera as the sole input sensor and incrementally builds a dense surfel representations of the scene’s 3D geometry. We extend an existing hybrid sparse-dense monocular SLAM system, reformulating the mapping pipeline in terms of the Kannala-Brandt fisheye camera model. Each frame is processed in its original undistorted fisheye form, with no attempt to remove distortion. To estimate depth, we introduce a new version of the PackNet depth estimation neural network adapted for fisheye inputs. We reformulate PackNet’s multi-view stereo self-supervised loss in terms of the Kannala-Brandt fisheye camera model. To encourage the network to learn metric depth during training, the pose network is weakly supervised with the camera’s ground-truth inter-frame velocity. To improve overall performance, we additionally provide sparse depth supervision from dataset LiDAR and SICK laser scanners. We demonstrate our system’s performance on the realworld KITTI-360 benchmark dataset. Our experimental results show that our system is capable of accurate, metric camera tracking and dense surface reconstruction within local windows. Our system operates within real-time processing rates and in challenging conditions. We direct the reader to the following video where the system can be seen in operation: https://youtu.be/Y-9q_wfqocs.