Interactive digital twins enabling responsible extended reality applications

Digital twins are living digital surrogates of objects and processes from the world around us and serve many purposes in situations where a physical model cannot be used. Digital twins can be considered to still be in their nascent phase, be it in the variety of application domains or in the science of creating and manipulating them. The DIDYMOS-XR project uses digital twins to create large-scale (city-wide) and long-term eXtended Reality (XR) applications. In this paper, we propose three use cases—city planning, city maintenance, and city tourism—along with an industrial use case for manufacturing environments, demonstrated through evaluation sites in Germany and Spain. The paper introduces cutting-edge solutions in the creation, simulation, and manipulation of DTs, including advancements in 3D scene reconstruction, data enhancement, data compression, sensor fusion, localization, rendering, and scene understanding. Additionally, we address the ethical and privacy challenges of digital twin systems and propose strategies for mitigating these issues. This work offers a comprehensive framework for creating semantically rich, scalable, and interactive digital twins, providing valuable insights for diverse applications in urban planning, maintenance, tourism, and industrial optimization.