Enabling data-driven human digital twin (HDT) reconstruction with computer vision and sensor-based positional data

Human Digital Twins (HDT) technology is rapidly emerging as a cornerstone of Industry 5.0, enabling personnel training, ergonomic analysis, and AR/VR applications. However, reconstructing accurate HDTs faces significant challenges, including difficulty in capturing complex human postures, susceptibility to environmental noise, occlusion problems, and synchronisation issues between multiple data sources. While positional tracking systems provide highly accurate localised measurements, they lack comprehensive coverage of full-body kinematics. In contrast, Computer Vision (CV)-based methods offer holistic body reconstruction but suffer from reduced accuracy, particularly in suboptimal lighting, unfavourable camera angles, or obstructed environments. Similar studies have only been applied in constrained setups or rely heavily on deep learning regressors, which can limit generalisability; in contrast, our work provides a more flexible and broadly applicable HDT framework, well adapted to general XR scenarios and readily extensible to other applications. This paper introduces a multi-modal HDT reconstruction framework that integrates CV-based body estimation with sensor-based positional tracking. By employing advanced filtering techniques, temporal alignment strategies, and data fusion pipelines, the framework leverages the complementary strengths of both modalities. The resulting HDTs are robust to environmental noise, scalable across scenarios, and sufficiently accurate for safety-critical applications. A case study in civil and construction training demonstrates how the proposed approach enables reliable behaviour capture and visualisation in XR environments. This work demonstrates the feasibility of multi-modal HDTs for enhancing training and safety in industrial environments.