Pose Estimation in Indian Sculptures using Hierarchical Assembly of Keypoints

Pose (or posture) of human figures in various art forms, including sculptures, provide significant cues for their interpretation and appreciation of the theme depicted therein. While human pose estimation in various contexts is an established domain in computer vision, there is a lack of work on pose estimation of human figures in art forms, especially of sculptures. Deep models trained for human pose estimation are seen to not work as well on human figure in art forms. The lack of work in this area is possibly symptomatic of a lack of annotated pose datasets of human figures in art forms which is necessary to retrain and fine tune these models. We present here, a new dataset of more than 9000 images of Indian sculptures which have been manually annotated with pose information for sculptures containing humanoid forms. Pose estimation in Indian sculptures presents a unique challenge as several deities are depicted with multiple limbs and heads, markedly different from a typical human anatomy. This makes the current pose estimation models unusable as they presume a standard human anatomy with fixed number of joints (keypoints) and represent person instances as a flat array of keypoints. We propose a new model with hierarchical assembly of keypoints as a tree, rather than a flat array. This reformulation of keypoint assembly retains rich information of keypoint pair connections in the final output and accommodates the possibility of multiple limbs of the same type. We also present a new similarity metric that generalizes the commonly used Object Keypoint Similarity (OKS) metric, and is suitable for ”tree-like” instances. Lastly, we highlight the relevance of proposed method of hierarchical assembly of keypoints for other areas in computer vision beyond pose estimation of sculptures.