Robust full‐motion recovery of head by dynamic templates and re‐registration techniques

Abstract This article presents a method to recover the full‐motion (3 rotations and 3 translations) of the head from an input video using a cylindrical head model. Given an initial reference template of the head image and the corresponding head pose, the head model is created and full head motion is recovered automatically. The robustness of the approach is achieved by a combination of three techniques. First, we use the iteratively reweighted least squares (IRLS) technique in conjunction with the image gradient to accommodate nonrigid motion and occlusion. Second, while tracking, the templates are dynamically updated to diminish the effects of self‐occlusion and gradual lighting changes and to maintain accurate tracking even when the face moves out of view of the camera. Third, to minimize error accumulation inherent in the use of dynamic templates, we re‐register images to a reference template whenever head pose is close to that in the template. The performance of the method, which runs in real time, was evaluated in three separate experiments using image sequences (both synthetic and real) for which ground truth head motion was known. The real sequences included pitch and yaw as large as 40° and 75°, respectively. The average recovery accuracy of the 3D rotations was about 3°. In a further test, the method was used as part of a facial expression analysis system intended for use with spontaneous facial behavior in which moderate head motion is common. Image data consisted of 1‐min of video from each of 10 subjects while engaged in a two‐person interview. The method successfully stabilized face and eye images allowing for 98% accuracy in automatic blink recognition. © 2003 Wiley Periodicals, Inc. Int J Imaging Syst Technol 13: 85–94, 2003; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ima.10048