Automatic Detection of Unsafe Behaviors Resulting from the Improper Use of PPE during Human–Robot Collaboration

This is the authors' abstract. We don't add key points for this paper.

The increasing integration of robotics in the industrial and construction sectors has significantly enhanced productivity and efficiency. However, the rapid adoption of human–robot collaboration (HRC) introduces unique safety challenges, particularly regarding workers’ improper use of personal protective equipment (PPE). These issues, if not effectively addressed, can compromise safety and impede the widespread adoption of robotic technologies. This study proposes an automated framework that combines semantic scene graph generation and named entity recognition (NER) to detect unsafe behaviors in real time across diverse HRC scenarios. The method employs encoder-decoder-attention (EDA) architectures and natural language processing techniques to transform unstructured visual data into structured textual descriptions. By integrating domain-specific safety knowledge, the framework enables accurate hazard inference and actionable insights for safety management. For HRC image understanding, the EDA achieved superior results with a convergence loss of 1.089, 99.1% accuracy, and a Bilingual Evaluation Understudy (BLEU)–4 score of 0.8905. Then the detection precisions for entities like human, behavior, unsafe behavior, and robot are 0.84, 0.81, 0.78, and 0.79, respectively, with an overall mean precision of 0.81. Experimental results confirm the robustness and scalability of the proposed approach within HRC contexts, particularly in detecting unsafe behaviors arising from the improper or missing use of PPE. This study contributes to the development of intelligent safety monitoring systems, offering practical solutions to enhance safety compliance and foster sustainable HRC in complex industrial environments.