Deep Learning based Human Detection using Thermal-RGB Data Fusion for Safe Automotive Guided-Driving

Every year, the number of drivers increases, resulting in a corresponding increase in traffic fatalities. In Korea, pedestrian accidents constituted 35.5% of all traffic accidents for the last 2 years, with the number of child accidents rising annually. Autonomous vehicles currently rely on a lidar, which is insufficient in preventing accidents as it only recognizes obstacles that are far away. To mitigate these accidents, we propose selective thermal imaging data to identify people beyond the limited field of view. First, RGB camera image data for object recognition is performed. When vehicles or obstacles are present, the optional use of thermal data is applied. Thermal data can only identify a person, and it is used to prevent unforeseen incidents. The RGB images are divided into thirds and each section is assessed for obstacles, prioritizing the area with the most obstacles for integration with thermal data. Using the described algorithm, the level of accuracy increased by 2.07 times, from 40.43% to 83.91%. Additionally, experiments performed on a personal computer demonstrate that the algorithm is capable of functioning in real-time at a rate of 2.7 frames per second, utilizing 175.95 megabytes of memory at 0.36 seconds per image. When executed on a lightweight board such as the Jetson Nano, the algorithm runs at a rate of 0.75 frames per second, utilizing 140.08 megabytes of memory, at 1.33 seconds per image.