Health-Critical Event Detection in Hajj Crowds Using Action Recognition and Facial Affect Analysis

Crowd anomaly detection is a critical capability for public safety in large-scale gatherings. Although a wide range of anomaly-detection methods has been proposed, most existing work concentrates on motion irregularities and crowd-density estimation. In contrast, comparatively little attention has been given to detecting individual-level, health-related indicators. This motivates us in the paper to present a computer-vision-based monitoring framework for detecting health-critical behaviors and early distress cues in large-scale pilgrimage crowds, using object detection and facial affect analysis. As existing datasets lack a specific focus on health-related indicators, we defined nine health-focused classes and developed Hajj-Health dataset. This dataset extends HAJJv2 by incorporating relevant objects from it, alongside newly collected and AI-generated images representing two additional classes: Raise Hand for Help and Receiving Medical Assistance, to capture health-related behaviors during Hajj. Different YOLO versions, including YOLOv8, YOLOv9, YOLOv10, and YOLOv11, were trained and evaluated for action recognition. The experimental results demonstrate that YOLOv8 and YOLOv9 achieve the overall highest performance in terms of precision, recall and mAP. In addition, the framework detects faces and identifies stress-related affect using RetinaFace for face detection and DeepFace for emotion recognition. These findings indicate that the proposed approach can reliably detect standard crowd behaviors, subtle health indicators, and stress-related cues, supporting early intervention in dense gatherings. This proposed framework lays the foundation for real-time, health-aware crowd monitoring during Hajj and similar mass events within cognitive city environments.