Markerless real-time garment retexturing from monocular 3D reconstruction

We present a fusion of augmented reality (AR) and virtual try on (VTO) that incorporates sparse 3D point recovery by exploiting distance constraints based on 2D point correspondences between a deformed texture in monocular video and a reference texture which is derived from the start of the sequence by face detection aided segmentation. A hierarchical and multi-resolution surface reconstruction approach is proposed, employing thin-plate splines, cloth modeling, and patch tessellation. Our method attempts to accurately recover a rectangular surface from a deformed arbitrarily shaped texture. We also propose a hue-based method for segmenting garment cloth and printed texture. The methods are demonstrated in an AR framework for real-time visualization of a virtual garment worn in a real scene. Real-time AR cloth retexturing from monocular vision is a state-of-the-art field. Previous work lacks realism and accuracy, only recovering the 2D cloth layout and lacks robustness, requiring a special T-shirt color and simple texture along with lab hardware. Our approach alleviates these limitations. We design a practical approach which considers a typical consumer environment with a mid-range PC and webcam. Our results are convincing and photorealistic with robustness to arbitrary T-shirts, subjects, and backgrounds. Future work will focus on extending our global model and quantitative analysis.