Towards socially acceptable XR agents: A modular approach for user interaction

Recent advancements in eXtended Reality (XR) technologies have opened new opportunities for integrating virtual and physical environments, enabling natural immersive user experiences. This paper addresses the challenge of developing socially acceptable XR agents that can engage in complex, meaningful interactions across various social and public settings. Specifically, we propose a modular architecture for enhancing user-XR agent interaction, integrated into the framework of the Horizon Europe project “Socially-acceptable Extended Reality Models And Systems (SERMAS)”. The architecture is designed to ensure adaptability, scalability, and the integration of diverse communication modalities, including verbal and non-verbal cues. Its main components work together to enable seamless user detection and communication. In particular, the Detection module plays a key role in managing the agent’s awareness of the surrounding environment. Its functions include identifying user intentions, monitoring group dynamics, and inferring emotional states, thereby increasing responsiveness and enabling more contextually grounded behavior. In contrast, the Communication module employs Large Language Models for personalized verbal responses and recognizes gestures and body language for enriched non-verbal communication. Together, these approaches guarantee contextually relevant, personalized, and emotionally intelligent interactions. The architecture supports scalability and the flexible integration or replacement of modules, fostering socially acceptable user-XR agent interactions. Its real-world applicability is demonstrated across three deployment scenarios, including a digital receptionist integrated with a physical robot, an XR-based security training system for journalists practicing safety-critical procedures, and a digital assistant supporting customers in a post-office environment. The functionality of the proposed system is presented through the implementation of the XR Agent interacting with users.