Lightweight Multi-Feature Fusion GAN with Deformable Attention for HMD-Occluded Face Reconstruction

Head-mounted displays (HMDs) enhance virtual reality (VR) experiences, but occlude the upper face, hindering realistic user representation. To address this, some studies employ sensors to capture facial expressions under occlusion, while deep learning methods typically rely on image inpainting to restore missing regions. However, these approaches often suffer from limitations such as insufficient shallow feature representation, high computational complexity, and redundant model structures. This study proposes a lightweight generative adversarial network (GAN) that utilizes multi-feature fusion and deformable attention for face reconstruction under HMD occlusion. Specifically, a Lie group feature learning module is used to enhance shallow geometric representations, while reference-guided deformable attention dynamically focuses on occluded regions, improving both structural fidelity and efficiency. Experiments across multiple face datasets show that the proposed method outperforms existing mainstream approaches regarding structural fidelity, detail restoration capability, and model efficiency. The proposed framework offers a promising solution for integration with HMDs equipped with facial tracking, enabling more realistic and expressive avatars in VR applications.