Novel hybrid multi-atlas and deep network framework with generalizability for skull base neural foramina segmentation

Abstract Accurate and complete segmentation of neural foramina, including the foramen ovale (FO) and foramen rotundum (FR) in the skull base, is essential for CT-guided percutaneous puncture targeting the trigeminal ganglion. Existing methods often focus on the internal cavity, resulting in incomplete anatomical delineation and insufficient accuracy for clinical application. This study aims to develop and validate a novel, generalizable segmentation method for the robust and precise segmentation of the complete bony structures of neural foramina. We propose an innovative hybrid multi-atlas and deep network framework. This framework first introduces an adaptive atlas selection strategy based on Otsu’s method to optimize the quality of fused atlas. For label fusion, we design a novel weight estimation network that combines a customized soft attention mechanism with a Mamba module to enhance feature representation and model long-range dependencies. The proposed method is rigorously evaluated via cross-validation on the atlas library and independent generalizability tests on unseen datasets. The proposed method demonstrated superior performance. On the atlas library, it achieved ASSD of 0.22/0.19 mm, 95HD of 1.54/1.55 mm, and DSC of 0.82 and 0.82 for the FO and FR, respectively. On the unseen dataset, the corresponding metrics were 0.32/0.32 mm for ASSD, 1.82/2.36 mm for 95HD, and 0.78/0.77 for DSC. The framework remained highly competitive performance against state-of-the-art methods. The proposed method achieves leading accuracy while maintaining high anatomical integrity. Its competitive performance on unseen datasets highlights strong generalizability and potential applicability in neural puncture surgery.