Heading Estimation of Ships using Deep Learning and Synthetic Non-aerial Images

In autonomous vehicle systems, it is paramount to estimate the pose of other vehicles in the surroundings to predict their motion and navigate efficiently and safely. In many situations, such as when distant ships are being tracked, camera images may be the only source of high-resolution information available for estimating the orientation of the target. Under these circumstances, we present a deep learning network for heading estimation of maritime surface vessels from a single non-aerial image. The literature on vehicle pose estimation using deep learning methods is dominated by automotive applications. Many of these applications use parametric three-dimensional models of cars, known limits on their dimensions, the assumption that roll and pitch angles are zero, and even traffic markings to infer the pose of the vehicle. Similar properties do not exist in maritime environments, mainly because ships vary greatly in shape and size, and their motion, in principle, has all six degrees of freedom. Hence, the orientation of the ship must be determined exclusively by the features present in the image. The proposed network consists mainly of two branches that build on a pre-trained deep convolutional neural network for feature extraction. The first branch of layers constitutes a classification network, and it provides a coarse estimate of the heading by dividing all possible directions on the horizontal plane into sector classes, while the second branch calculates a refinement of the heading angle, giving the final estimate. Furthermore, a dataset of synthetic non-aerial images of ships is generated using three-dimensional rendering software and used for training and testing the network. Finally, the network is refined by transfer learning using a collection of real-world images of ships. Although conceptually simple, the network outperforms similar methods that use low-resolution information such as radar.