PEC-Moni: learning to monitor compliance of personal protective equipment usage using joint 2D-3D representation for intelligent laboratory information management

Recent advancements in computer vision have significantly improved intelligent Laboratory Information Management System (LIMS). However, challenges remain in accurately detecting the usage compliance of Personal Protective Equipment (PPE) due to variations in human posture and environmental conditions that result in unsatisfactory generalization capacity in new laboratory scenarios. On the other hand, existing methods model the PPE usage compliance into an object detection problem and employ current detectors in 2D images to solve it, such as YOLO and DETR, which might hold information loss issues with local information ignorance and single modality consideration. In this paper, we propose PEC-Moni , a novel framework designed for PPE compliance monitoring task with joint 2D-3D representation. Specifically, PEC-Moni firstly segments and extracts skeleton for each worker instance from 2D images. Then we learn a 2D-3D projection function that maps the segmented points into 3D SMPL point clouds. Meanwhile, we utilize the 2D keypoints to obtain the face, hand and body regions with SAM model, and combine with the 3D point clouds to judge the final PPE compliance result. We evaluate PEC-Moni on our own built benchmark PEC-Bench and provide extensive experiments. The results show that PEC-Moni achieves state-of-the-art performance in PPE compliance monitoring task. We believe our approach has the potential for various LIMS applications across various domains, including PPE monitoring heatmap analysis and person-object relation graph generation. The source code and dataset will be publicly available.