Three‐stage RGBD architecture for vehicle and pedestrian detection using convolutional neural networks and stereo vision

With the growth of autonomous vehicles and collision‐avoidance systems, several approaches using deep learning and convolutional neural networks (CNNs) continually address accuracy improvement in obstacle detection. The authors introduce a three‐stage architecture that adds side channels as low‐level features to serve as input to existing CNNs. In a case study, the architecture is used to extract depth from stereo cameras, and then compose RGBD inputs to state‐of‐the‐art CNNs to improve their vehicle and pedestrian detection accuracy. This can be achieved by simple modifications on the first layers of any existing CNN with RGB inputs. To validate the architecture, the state‐of‐the‐art matching cost‐CNN, and cascade residual learning, both specialist algorithms to extract depth information combined to the state‐of‐the‐art Faster‐region‐based CNN, MSCNCN, and Subcategory‐aware Convolutional Neural Network (SubCNN) to yield the models to be tested using the KITTI dataset benchmark. In many cases, the accuracy (in terms of average precision) using their proposal outperforms the original scores in various scenarios of detection difficulty, reaching improvements up to +3.96% in the training and +1.50% in the testing KITTI datasets. This proposal also introduces efficient methods to initialise the weights of the depth convolutional filters during transfer learning using net surgery.