Reliable Object Pose Alignment in Mixed-Reality Environments Using Background-Referenced 3D Reconstruction

Accurate alignment of real-world object poses with their virtual counterparts using sensors, e.g. cameras, is essential for consistent interaction in mixed-reality systems. However, objects can undergo abrupt, untracked movements during periods when a tracking system is inactive, e.g., overnight, causing stored pose records to become inconsistent with the real scene and breaking user interaction in the virtual environment. Off-the-shelf 3D reconstruction networks such as MASt3R (Matching and Stereo 3D Reconstruction) method provide metrically scaled 3D point maps and pixel correspondences, but they are trained on static scenes and therefore fail to produce reliable object correspondences when the object has moved. We propose a robust pipeline that combines MASt3R’s metrically scaled 3D outputs with a background-based alignment strategy to recover and apply the true pose change of moved objects. Our method first segments foreground and background and extracts 3D background point sets for a reference day and a current day. An affine transformation between these background point sets is estimated via a standard registration technique and used to express the current-day object 3D coordinates in the reference coordinate frame. Within that unified frame we compute the object pose change and apply the resulting transform to the virtual object, restoring real–virtual consistency. Experiments on real scenes demonstrate that the proposed approach reliably corrects pose misalignments introduced during inactive periods and substantially improves over applying MASt3R alone, thereby enabling restored and consistent user interaction in the virtual environment.