A Multimodal Software Architecture for Human Digital Twins in Human-Machine Interaction Design and Proof-of-Concept Evaluation

This is the authors' abstract. We don't add key points for this paper.

Human Digital Twins require software architecturesthat can integrate heterogeneous data sources while preservinginteroperability, traceability, synchronization, and controlledevolution. This paper proposes a multimodal software architecturefor Human Digital Twins in human-machine interaction scenarios.The proposal was derived from a literature-based analysisand refined through Attribute Driven Design. The architectureorganizes the solution into layers for multimodal ingestion,normalization and synchronization, integration and orchestration,Human Digital Twin services, persistence and traceability, andexternal consumption. A prototype was evaluated through sixcontrolled experiments and a second evaluation stage with selectedsubsets of the WESAD and HA4M datasets. The controlledexperiments covered nominal operation, high load, alarm behavior,video extension, degraded capture, and noisy physiological data.WESAD introduced real physiological signals while preservingthe two-modality baseline of 160 persisted events, eight windows,eight snapshots, and no incidents. HA4M was evaluated in twostages: traceable references to real RGB frames and simple framederivedindicators. Its final stage preserved the three-modalitybaseline of 240 events, eight windows, eight snapshots, and noincidents. The results indicate that the architecture can supportmultimodal integration, real-data source adaptation, localizedextension, and operational continuity within the evaluated scope.