Deep Learning for Enhanced 3D Landmark Positioning: Addressing Biases in Traditional Registration Methods

Geometric morphometrics has become a widely used toolkit for quantitative morphology across diverse biological fields. It relies on the precise placement of anatomical landmarks in 2D or 3D. However, manual landmarking, especially in 3D, is labour-intensive, time-consuming, prone to intra- and inter-observer variation and requires significant expertise. These limitations are increasingly incompatible with the demands of high-throughput imaging workflows. This bottleneck has spurred efforts to automate landmarking, often through deformable registration of 3D images, surfaces, or point clouds. While these methods can effectively capture overall shape, they may introduce biases in landmark localization and distort patterns of variances and covariances. Previous attempts to mitigate these issues with deep learning (DL) have generally focused on specific anatomical parts of certain species, most notably human faces and craniofacial anatomy in model-organisms. In this study, we introduce a DL-based pipeline for refining landmark placement. The pipeline begins with approximate landmark predictions derived from a simple rigid registration. These initial predictions define local 3D surface regions, which are then parametrized into 2D using least-squares conformal maps and enhanced with colorization informed by diverse geometry and ambient lighting. The resulting standardized representations are then used to train Transformer and Convolutional Neural Network (CNN) architectures. The approach was evaluated using an open-access dataset of 3D mouse skull models with manually digitized landmarks. Focusing on ten distinct landmarks, we compared our rigid + DL predictions with those from a mainstream global registration approach, ALPACA. We assessed both distance accuracy to the manual groundtruth and preservation of the shape variation patterns. We also examined the effects of various hyperparameter settings strategies, offering practical guidelines for future implementations. Our results demonstrate the feasibility of using 2D colored-enhanced local geometric representations for accurate landmark position refinement with DL. The proposed approach mitigates biases associated with the considered deformable registration method and establishes a methodological foundation for more generalizable, high-throughput morphometric analyses using DL.