Detection and Semantic Segmentation of Disaster Damage in UAV Footage

In the aftermath of large-scale disasters, such as hurricanes, floods, or earthquakes, preliminarily damage assessment (PDA) is carried out to determine the impact and magnitude of damage and meet the needs of affected individuals, businesses, and communities. Traditionally, site evaluation and consensus-based assessment techniques are used to estimate the extent of the damage. More recently, given their low-cost, ease of operation, and ability to be deployed ondemand, unmanned aerial vehicles (UAVs) are increasingly used for disaster response and mitigation. However, the resulting large volume of visual data collected by and shared among first responders and volunteer groups is not used effectively because current practices of processing such data are heavily human-dependent, extremely resource-intensive, and significantly slow compared to the fast-evolving nature and progression of disaster impact. This paper contributes to the core body of knowledge by presenting a fully annotated dataset (with the object classes people, flooded area, and damaged and undamaged building roof, car, debris, vegetation, road, and boat) and a host of convolutional neural network (CNN) models for detecting and segmenting critical objects in the aerial footage of disaster sites. For best results, two CNN-based image segmentation architectures, namely, Mask-RCNN and Pyramid Scene Parsing Network (PSPNet), are adopted (through transfer learning), trained, validated, and tested on annotated videos to detect countable and bulk objects. The paper further introduces a targeted data augmentation technique to preserve data balance, as well as a data-driven approach to splitting highly mismatched classes for better model performance. Through these improvements, the best performing Mask-RCNN model generates pixel-level segmentations of countable objects with a 51.54% mean average precision (mAP). Additionally, the best performing PSPNet models can achieve mean intersection over union (mIoU) as high as 32.17% and accuracy as high as 77.01% on bulk objects.