Autonomous port traffic safety orientated vehicle kinematic information exploitation via port-like videos

Abstract This study proposes a framework for extracting automatic guided vehicle (AGV) kinematic information from port-like videos, which provides a solution for situation awareness of port surveillance videos. Firstly, vehicle pixel-wise positions in port-like videos are determined by the visual object tracking (SeqTrack) model. Secondly, the extrinsic parameters of the query images are estimated by the generalizable model-free 6-DoF object (Gen6D) pose estimation method. More specifically, a point cloud of AGV is reconstructed with multi-view AGV reference images and the image extrinsic parameters are obtained through structure form motion. The reference image which has most similar viewpoint with query image is identified with the Gen6D selection module, as a result, the extrinsic parameters of the reference image can be used to estimate the extrinsic parameters of the query image. After that, extrinsic parameters of query image are identified with the support of Gen6D refinement module. Thirdly, we obtain vehicle displacement by mapping the vehicle point cloud coordinate into the camera coordinate, and then we estimate vehicle movement information with the help of generative adversarial network (GAN) model. Experimental results suggest that the pose estimation metrics average discrepancy distance and 2D re-projection error of our method reach 0.76 and 0.75, respectively. The mean absolute error (MAE) and root mean squared error (RMSE) of the estimated vehicle displacement reach 0.023, 0.030 for scene #1 (i.e. AGV moves along x-axis of camera coordinate) and 0.182, 0.298 for scene #2 (i.e. camera follows AGV to move along x-axis of camera coordinate).