Precise cell recovery by cell nucleus united transcript (CellCUT) for enhanced spatial transcriptomics

Abstract Cell segmentation is the first step in parsing spatial transcriptomic data, often a challenging task. Existing cell segmentation methods do not fully leverage spatial cues between nuclear images and transcripts, tending to produce undesirable cell profiles for densely packed cells. Here, we propose CellCUT to perform cell segmentation and transcript assignment without additional manual annotations. CellCUT provides a flexible computational framework that maintains high segmentation accuracy across diverse tissues and spatial transcriptomics protocols, showing superior capabilities compared to state-of-the-art methods. CellCUT is a robust model to deal with undesirable data such as low contrast intensity, localized absence of transcripts, and blurred images. CellCUT supports a human-in-the-loop workflow to enhance its generalizability to customized datasets. CellCUT identifies subcellular structures, enabling insights at both the single-cell and subcellular levels.