Machine Learning-Based Risk Analysis for Construction Worker Safety from Ubiquitous Site Photos and Videos

This paper proposes a new method for single-worker severity level prediction from already collected site images and video clips. Onsite safety observers often assess workers’ severity levels during construction activities. While risk analysis is key to improving long–term construction site safety, omnipresent monitoring is still time-consuming and costly to implement. The recent growth of visual data captured actively on construction sites has opened a new opportunity to increase the frequency of worker safety monitoring. This paper shows that a comprehensive vision-based assessment is the most informative to automatically infer worker severity level from images. Efficient computer vision models are presented to conduct this risk analysis. The method is validated on a challenging image dataset first of its kind. Specifically, the proposed method detects and evaluates the worker state from visual data, defined by (1) worker body posture, (2) the usage of personal protective equipment, (3) worker interactions with tools and materials, (4) the construction activity being performed, and (5) the presence of surrounding workplace hazards. To estimate the worker state, a multitasked recognition model is introduced that recognizes objects, activity, and keypoints from visual data simultaneously, taking 36.6% less time and 40.1% less memory while keeping comparably performances compared to a system running individual models for each subtask. Worker activity recognition is further improved with a spatio-temporal graph neural network model using recognized per-frame worker activity, detected bounding boxes of tools and materials, and estimated worker poses. Finally, severity levels are predicted by a trained classifier on a dataset of images of construction workers accompanied with ground truth severity level annotations. In the test dataset assembled from real-world projects, the severity level prediction model achieves 85.7% cross-validation accuracy in a bricklaying task and 86.6% cross-validation accuracy for a plastering task, demonstrating the potential for near real-time worker safety detection and severity assessment.