Special Issue: Extended Reality in Design and Manufacturing

Extended Reality (XR) is a collective term that contains Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), and everything in between. VR is an immersive technology that allows users to interact within a computer-generated 3D virtual environment. AR enhances the user's situational awareness (SA) by superimposing virtual content onto a real scene. MR is a step beyond AR by combining physical reality with digital content in a way that enables real-time interaction with and among real and virtual elements. In the business world, a similarly captivating term, metaverse, indicates a single, universal, and immersive virtual world focused on social and economic connections facilitated by the use of XR technologies [1].XR has proven to be an effective communication medium for humans to instantly interact with other entities in shared spaces. As a result, it has received significant attention from both industry and academia. VR/AR is among the nine enabling technologies in the Industry 4.0 paradigm [2], serving as the only human-centric technology designed specifically for end users. This feature conforms well to the human-centric core value emphasized by the emerging Industry 5.0 [3]. With this advantage, XR could play a critical role in extending cyber-physical systems (CPS) into human-cyber-physical systems (HCPS), seamlessly integrating data, computational intelligence, and real-world situations through context-specific sensory cues for human interaction.With the recent advances in information and communication technologies (ICT), traditional manufacturing systems have evolved into “smart” ones in handling the ever-increasing complexity of tasks incorporating a high degree of autonomy and intelligence. Nevertheless, there are specific tasks on the shop floor that are either impractical or impossible to complete without human intervention. In such situations, the main purpose of smart manufacturing would be to facilitate manual operations, rather than to completely replace human involvement. Indeed, integrated with artificial intelligence (AI), 5G, cloud computing, and Internet-of-Things (IoT), XR applications have been successfully implemented and deployed to assist a diverse range of manual operations within the manufacturing sector [4,5]. The recent special issue titled “Challenges and Opportunities in Computing Research to Enable Next Generation Engineering Applications” published in the Journal of Computing and Information Science in Engineering (JCISE) also features three review articles focusing on extended reality [6–8]. In addition, the journal has published 16 papers with the words VR, AR, MR, XR, or Metaverse appearing in either the title or abstract between 2018 and 2023 [9–24]. This trend shows a growing interest in XR research in the JCISE community. A special issue focusing on the related fields would be necessary and timely to address this need.The purpose of this special issue is to solicit articles describing fundamental and applied research on XR techniques, solutions, and challenges in the area of design and manufacturing. This issue aims to cover all aspects of XR technology, including technical advances in software/hardware, algorithms, ergonomic analysis, system implementation, and novel empirical studies, for product design and manufacturing processes. In total, eight papers are included following rigorous review and revisions, and each is briefly introduced as follows.AR technology has made significant progress in recent years with applications successfully applied to assist manual tasks in various industries such as manufacturing, military, education, and entertainment. For instance, manual pointing and trajectory tracing are often used in robot programming in the manufacturing environment or during major surgical operations. Design guidelines are essential to develop assisted functions that can effectively enhance users’ situational awareness with instant sensory feedback through AR interfaces. For this purpose, Wang et al. conducted an experimental study on AR user interfaces with various sensory cues designed to assist in the manual tracing of simple and complex trajectories. The study compared the task performance under five sensory feedback conditions: central vision, peripheral vision, auditory sensation, force feedback, and no feedback. The experimental results showed that the feedback information should not only provide prompts for error status but also offer suggestions for correcting the actions of the subjects in complex trajectory tracing. The force feedback demonstrated better adaptation compared to visual and auditory cues. Furthermore, subjects tended to rely more on real-time force feedback to complete difficult tracing tasks than on the other cues. It was also suggested that the design of AR-assisted functions and instant sensory feedback should consider the potential limitations induced by the user's posture during a tracing task. These experimental findings could provide useful insights into the design of sensory feedback for AR-assisted functions developed for manual tracing tasks.Fewer studies have focused on the technical solutions for XR-based virtual footwear try-ons compared to those for garments and facial accessories. Therefore, Chou et al. introduced an approach based on machine learning that estimates the 6-DoF of a human foot using a single RGB image captured from a third-person view. Synthetic annotated data were generated by systematically adjusting the light source, foot geometry, background, camera position, as well as skin and background textures to ensure sufficient variation in information during the training process. Three CNN (Convolutional Neural Network) models (DOPE, DOPE2, and YOLO-6D) were trained using a proprietary dataset containing both synthetic and real foot images, following different training plans. Test results showed that training a CNN model for foot pose estimation performed better through a combination of pretraining and successive training with both proprietary and ImageNet datasets. In real use, YOLO-6D outperformed both the DOPE and simplified DOPE2 models in terms of prediction accuracy and computational speed.Natural human–computer interaction (HCI) remains an important research topic for industrial XR. Three articles featured in this special issue explore HCI from the perspectives of content authoring, cybersecurity, and human pose estimation. Content authoring for XR applications remains a challenge at scale. In response, Ipsita et al. developed VRFromX, an application-embedded workflow for in situ content creation and demonstrated its utility through a welding use case. Additional research is warranted to properly integrate XR technologies with minimal disruption into existing practical engineering workflows, such as the management of computer-numerical control (CNC) machines within a factory.Yang et al. addressed a potential cybersecurity risk existing in mid-air interactions commonly used in MR-based smart manufacturing applications. The authors developed a prototyping system for smart factory management using a video see-through MR HMD. This system allows users to input a password on a login interface, providing access to security/privacy-sensitive information through a single RGB camera capturing mid-air gestures in front of the user. A machine learning-based inference framework tracks hand gestures and estimates input passwords under assumed attack conditions. The test results in two scenarios suggested that potential preventive measures for cybersecurity attacks in MR applications during password input with hand gestures include using more complex characters, applying remote interaction methods, or entering the password while moving.The AR content presented by the head-mounted display (HMD) offers mobility, hands-free capabilities, and a first-person view, particularly useful in the context of industrial applications. The successful deployment of AR applications on these occasions depends on humans accurately interacting with and placing virtual objects in relation to the physical environment. This requires precise alignment and human perception of computer-generated graphics within the physical environment. Yang and Zhang resolved the misregistration issues occurring in video see-through AR HMDs that limit the potential for shared workspaces between humans and machines in shop-floor applications. They identified and analyzed the sources of errors within a comprehensive system framework. A mathematical model was proposed to characterize the influence and sensitivity of errors specifically in the HMD-to-camera transformation. This was achieved by examining three translational and three rotational components comprising the transformation matrix and simulating the resulting misregistration effects. An offline closed-loop calibration approach was applied to reduce the HMD-to-camera transformation error in a prototyping system. Both qualitative and quantitative evaluations demonstrated satisfactory global registration between virtual objects and the physical environment.Wu et al. implemented ImpersonatAR, a mobile-based application, that utilizes embodied demonstrations in the augmented environment to support the prototyping and evaluation of multi-scenario AR use cases. The application was developed based on a set of design considerations derived from interviews with AR expert designers to address the current limitations of AR prototyping tools. ImpersonatAR enables capturing events or steps through embodied demonstrations using avatars and 3D animations. It facilitates the organization of these events and steps to create a multi-scenario experience, allowing stakeholders to explore the scenarios through interactive role-play with the prototypes. Positive feedback from ten users validated that ImpersonatAR promotes the exploration of diverse design possibilities for multi-scenario AR use cases through embodied representations of different scenarios. This work contributes to human–computer interaction by providing an effective prototyping approach for AR use cases.Advancements in VR offer great opportunities to create effective digital learning and training environments. Multi-user distributed participation in VR facilitates collaborative problem-solving (CPS) and learning-by-doing, both of which are critical professional skills for solving real-world problems. In addition, VR applications integrating with sensors allow the collection of in-process user behavioral data for adaptation or correction. Kim et al. presented a sensor-based multiplayer VR manufacturing simulation game that improved virtual hands-on learning experiences and served as a behavioral modeling method for monitoring the collaborative problem-solving (CPS) skills of participants. The Virtual Learning Factory was proposed to facilitate low-cost implementation of virtual hands-on experiences as effective as their real-world equivalents. A dynamic network behavioral modeling approach was developed to extract time-varying patterns of player actions using action-log data using action-log data commonly available in digital learning environments. The heterogeneous-behavioral CPS score was defined by the graph similarity between players. Experimental results demonstrated that the score successfully monitored user-specific collaborative CPS dynamics between players, potentially leading to appropriate interventions to improve learning outcomes. This work enhances the effectiveness of virtual reality and gamification in engineering education.Nguyen et al. developed an augmented reality system for the human-in-the-loop registration of robot-sensor coordinates in production environments. The system efficiently records and visualizes the pose-dependent quality of computing the robot-sensor transformation, providing human operators with intuitive assessments of how changes in the configuration space affect registration and overall system performance. The metrics proposed to evaluate the quality of point measurements demonstrated a decrease in both the accuracy and precision of the registration as the points were measured farther from the robot base. The AR interface intuitively shows that the registration accuracy depends on the location of the sensor within the workspace. It may enable in situ manual re-sampling for better points to be used in registration. This technical brief highlights the potential of AR as a human-in-the-loop interface for metrology applications.The creation of this special issue indicates that extended reality-related studies have shifted from applications designed for various industrial sectors in earlier times to efforts aimed at improving practical values through the proposal of new techniques, algorithms, or assessment methods. It is worth noting that XR research must certainly be conducted from a human-centric perspective, given that its deployment or use is, after all, designed for people, not machines. Due to hardware device limitations, the positional precision of current VR/AR applications may fail to directly meet manufacturing requirements. However, these applications should function effectively in assisting manual tasks by enhancing human situation awareness and characterizing human behavior through the instant capture of their actions or intentions. This necessitates in-depth studies in human–computer interaction, particularly from ergonomic and UX/UI perspectives, in the context of engineering applications. We expect to see an increasing trend in using XR as an interfacing technology for humans to collaborate with computational intelligence (including artificial intelligence, simulation, analytics, and machines) in real time through a wide spectrum of technical elements such as sensors, IoT, cloud computing, and 5G/6G. This trend would transform Industry 4.0 into the emerging Industry 5.0 by fostering human–machine collaboration in a broader context. Especially, the intersection between XR and the recently booming large language model (LLM), as well as other multi-modality AI models, is expected to provide new affordances for new directions of innovation and advancement in broad areas of design and manufacturing. XR studies addressing sustainability and resilience issues could yield a higher degree of innovation and scientific impact along the transformation process.The guest editors would like to thank all contributing authors for their outstanding work. We want to express our sincere gratitude to the reviewers for dedicating their valuable time and efforts in providing constructive comments to the authors during the review process. In particular, we highly appreciate Ms. Amy Suski for her editorial assistance and timely reminders. Finally, we would like to extend our appreciation to Prof. S. K. Gupta and Prof. Y. Wang, the past and current Editor-in-Chief of JCISE, for offering the opportunity and their support for this special issue.Distribution A. Approved for public release: Distribution unlimited under AFRL-2024-0392 determined on January 23, 2024.