Vision-aided inertial navigation with rolling-shutter cameras

In this paper, we focus on the problem of pose estimation using measurements from an inertial measurement unit and a rolling-shutter (RS) camera. The challenges posed by RS image capture are typically addressed by using approximate, low-dimensional representations of the camera motion. However, when the motion contains significant accelerations (common in small-scale systems) these representations can lead to loss of accuracy. By contrast, we here describe a different approach, which exploits the inertial measurements to avoid any assumptions on the nature of the trajectory. Instead of parameterizing the trajectory, our approach parameterizes the errors in the trajectory estimates by a low-dimensional model. A key advantage of this approach is that, by using prior knowledge about the estimation errors, it is possible to obtain upper bounds on the modeling inaccuracies incurred by different choices of the parameterization’s dimension. These bounds can provide guarantees for the performance of the method, and facilitate addressing the accuracy–efficiency tradeoff. This RS formulation is used in an extended-Kalman-filter estimator for localization in unknown environments. Our results demonstrate that the resulting algorithm outperforms prior work, in terms of accuracy and computational cost. Moreover, we demonstrate that the algorithm makes it possible to use low-cost consumer devices (i.e. smartphones) for high-precision navigation on multiple platforms.