SPICEiST: subcellular RNA pattern enhances cell clustering of imaging-based spatial transcriptomics

Abstract Background Imaging-based spatial transcriptomics (ST) enables the quantification of gene expression at single-cell resolution while preserving spatial context, but its utility is limited by small gene panels and challenges in accurate cell segmentation. To address these limitations, we present a graph autoencoder framework that integrates subcellular transcript distribution patterns with cell-level gene expression profiles for enhanced cell clustering in imaging-based ST (SPICEiST). Results The clustering performance of SPICEiST was systematically evaluated across several cancer datasets and gene panel sizes. The results demonstrate that SPICEiST consistently outperforms the conventional cell-level gene expression-based methods in distinguishing subtle differences in cell states, as measured by the number of cell clusters and clustering indices, such as the CHI and DBI. Moreover, the findings indicate that SPICEiST can further enhance the performance, even with advancements in cell segmentation, particularly for datasets with small gene panels. Overall, these improvements in cell clustering indices, CHI and DBI, were more pronounced in datasets with small gene panels of around 300 genes, in contrast to those with large panels containing over a thousand genes. Notably, SPICEiST also reveals more spatially intermixed and less compartmentalized cell clusters, a characteristic that better reflects the complex and heterogeneous nature of tumor microenvironments. This effect was especially evident in the datasets with large panels. Conclusions These findings highlight the value of leveraging subcellular transcript patterns to overcome the inherent limitations of imaging-based ST, particularly for small gene panels, and may provide new insights into tumor heterogeneity.