Deep Learning‐Based Quality Control Using Subcellular RNA Spatial Distribution Patterns for Cell Segmentation in Spatial Transcriptomics Data

Sequencing‐based spatial transcriptomics (sST) techniques with high resolution enable transcriptome‐wide RNA capture at subcellular resolution. Although new cell segmentation methods for sST data are continually being developed, accurately assigning RNA spots to corresponding cells still presents significant challenges and there is a lack of quality control methods. This work introduces a deep learning method for quality control of cell segmentation and improvement of the segmentation result. The proposed method exploits the subcellular spatial distribution patterns of different types of RNA by a deep neural network to assess the quality of segmented cells. The method identifies partially segmented cells typically due to low RNA capture or strong RNA diffusion as well as merged cells due to high cell density. In addition, the quality control method is combined with a Transformer‐based cell segmentation method and it is shown that the cell segmentation performance improves by automatically removing low‐quality segmented cells from the training dataset. The method is applied to both synthetic data and real Stereo‐seq data, demonstrating its potential for quality control and enhancement of cell segmentation in sST data.