This study focuses on building an interactive system that combines a virtual game (VG) and a physical robot (PR) to resolve cross-domain problems. These problems include the lack of precise matching between the virtual and physical components and the inadequate real-time features of the current integration. A system framework based on the client/server structure is built, and different functional components are allocated between the client and the server. The client handles image collection and camera calibration, while the server takes care of rendering the virtual scene, fusion of the virtual and physical scenes, and controlling the game robot. Optimization of the virtual–physical fusion demonstrates efficient hardware choice and software environments for development. Real-time synchronization of virtual and physical scene poses is realized by computing the camera’s internal and external parameters. This study designs an interactive module, helps communicate game robot parts, and handles login authentication and heartbeat packets. These features improve system performance, testing frame rate, responsiveness, and accuracy. The tests indicate no error in output frames in all four runs. The total frames increase with time, and the frame rate has a definite relation with the time used. If the sending time is different, the time difference is constant at around 4–9 microseconds. In addition, the system can blend virtual objects and the real environment well, even if the camera moves. If this system can be developed successfully, it can offer a practical technical solution to integrate PR and VG. Its excellent stability and accuracy pave the way for later real world industrial, educational, and rehabilitative applications, thus promoting the development of human–computer interaction technology.
He et al. (Sun,) studied this question.