Realistic Human Image Animation With Dynamic Camera Effects

Human image animation transfers a sequence of pose guidance to a character in an image to generate a dynamic video. In this paper, we enhance the realism of human image animation by introducing viewpoint-conditioned synthesis under moderate camera motion, enabling dynamic camera effects in addition to articuated human motion. However, simply employing off-the-shelf human image animation or view synthesis models in a cascaded manner leads to artifacts, because body motion and viewpoint transformation are handled independently wwithout preserving consistent depth ordering and occlusion. As such, we propose a method which jointly learns to perform both tasks within a shared 3D representation. Specifically, our framework obtains an inpainted background image and isolates the human figure from animated frames. Most importantly, we propose a background depth reconstruction module (BDRM), composed of a depth inpainter and a depth scaler, to obtain an aligned background depth. With the depth information, we convert the background and the foreground into 3D point clouds which are then rendered from target views. For the joint training, we suggest a T-shaped training method which resolves the time-consuming and costly process of collecting animated videos for each different view. In this paper, we prove the effectiveness of our background depth reconstruction module in preventing occlusions, and the superiority of our method over cascade approaches quantitatively and qualitatively. Our Project Page is available at https://excellent-wizard-cf6.notion. site/ieee-tcsvt-dynamic-camera-human-image-animation.