SPIDMA: skeletal-posture-based human intrusion detection with multimodal MR alerts for mechanical hazard zone

In industrial environments, human intrusion into mechanical hazard zone is a primary cause of occupational injuries. Accurately identifying such intrusions and implementing proactive warnings in complex scenarios remains a critical challenge in safety monitoring. This study proposes a proactive warning framework combining skeletal keypoint recognition and multimodal mixed reality (MR), named SPIDMA. This method extracts human skeletal keypoints from surveillance videos for posture analysis, dynamically adjusts the keypoint set used for intrusion judgment, and achieves high-precision detection utilizing a directed cross-product sign-flip logic relative to the warning line. To overcome the limitations of traditional alarms in visual blind spots and noisy environments, the system introduces a warning mechanism based on a mixed-reality head-mounted display (MR-HMD). Upon detecting an intrusion, it instantly delivers a multimodal alert integrating holographic visuals and spatial audio to operators, enabling immediate hazard avoidance. Experimental results demonstrate that SPIDMA achieves an accuracy, precision, recall, and F1-score of 89.52%, 88.61%, 90.00%, and 89.30%, respectively. Compared with the YOLO bounding-box center baseline, its accuracy and precision are relatively improved by 50.9% and 62.8%, respectively, significantly reducing false positives while maintaining high recall. Additionally, the end-to-end system latency is controlled between 211.6 ms and 286.4 ms, satisfying real-time operational requirements. SPIDMA enhances detection accuracy under complex postures and provides an instant warning response, offering a novel paradigm for proactive safety in industrial collaborative workspaces.