A Deep Learning-Based Pipeline for Segmenting the Cerebral Cortex Laminar Structure in Histology Images

Abstract Characterizing the anatomical structure and connectivity between cortical regions is a critical step towards understanding the information processing properties of the brain and will help provide insight into the nature of neurological disorders. A key feature of the mammalian cerebral cortex is its laminar structure, with the neocortex differentiated into up to six layers. Identifying these layers in neuroimaging data is important for providing a foundation for understanding the axonal projection patterns of neurons in the brain. These patterns can be seen in experiments using anterograde tracer or are reflected in the brain activity seen in layer-fMRI, etc. We studied Nissl-stained histological slice images of the brain of the common marmoset (Callithrix jacchus), which is a new world monkey that is becoming increasingly popular in the neuroscience community as an object of study. We present a novel computational framework that first acquired the cortical labels using AI-based tools followed by a trained deep learning model to segment cerebral cortical layers. We cross-tested and compared our pipeline with the existing advanced pipeline.