Shape from polarization based on a polarization representation and sparse self-attention

Accurate estimation of surface normals plays a crucial role in 3D reconstruction tasks. Physics-based shape from polarization (SfP) methods face numerous limitations, such as difficulty in handling mixed reflections and local ambiguities. In contrast, deep learning-based SfP methods surpass physics-based approaches in both accuracy and applicability. However, learning-based SfP methods still exhibit deficiencies in leveraging global context information and physical prior knowledge. To further enhance the accuracy of shape recovery, we propose what is believed to be a novel learning-based shape from polarization method that combines physical priors with sparse self-attention. First, we introduce the representation of polarization information based on Stokes vectors to enhance the efficient utilization of physical priors. Subsequently, we incorporate a sparse self-attention mechanism with bi-level routing to improve our network’s perception of global contextual information and better resolve local polarization ambiguities. Spatial and channel attention mechanisms are further employed to optimize feature fusion, enhancing the ability to capture high-frequency details. Finally, experiments conducted on the public DeepSfP dataset and the self-built dataset demonstrate that our method outperforms state-of-the-art SfP methods on all evaluation metrics and achieves a mean angular error of 12.06 ∘ on the DeepSfP dataset. By introducing a supposed novel polarization representation, sparse self-attention, and feature fusion techniques, our method significantly enhances the accuracy of normal estimation, providing promising technical support for the SfP task.