Real-time multi-worker identification and action recognition system employing multimodal deep learning for human digital twin

Abstract Worker safety in modern industrial environments is becoming increasingly critical owing to the complexities introduced by human–machine collaborations, making real-time activity recognition essential. Conventional unimodal human activity recognition (HAR), which relies solely on inertial sensors or cameras, often exhibits degraded performance in dynamic settings owing to occlusion, sensor faults, and poor scalability in multi-worker scenarios. To address these limitations, we propose a real-time multimodal HAR system as the core component of a human digital twin (HDT) for safety management. The system integrates inertial measurement unit (IMU) data from workers’ smartphones with vision data from existing closed-circuit television (CCTV) infrastructure, eliminating the need for additional wearables. Worker identity is maintained by linking smartphone identifiers with face recognition results obtained from video streams, enabling consistent tracking even when workers temporarily leave the camera’s field of view. An adaptive score fusion strategy dynamically balances modalities according to confidence, which improves the robustness of the system to missing or noisy data. This approach advances HAR beyond controlled laboratory settings and provides a practical and computationally efficient foundation for HDT-based safety management in industrial environments.