NCTDA: Nearest Neighbor Gaussian Process-Based Cell Type-Specific Spatially Variable Gene Detection Analysis

A primary task of spatial transcriptomics is detecting spatially variable genes (SVGs). Many genes may show spatially heterogeneous expression in specific cell types while showing spatial randomness across the whole tissue, thereby defining cell type-specific SVGs (ctSVGs). This study aims at detecting not only ctSVGs but also SVGs. Here, we construct a novel analysis framework NCTDA, which employs a nearest neighbor Gaussian process (NNGP) model to incorporate cell type composition into the spatial modeling of gene expression, is capable of linearly scaling with the number of spatial spots, unlike the cubic scalability of most methods. NCTDA performs hypothesis testing for different detection purposes, namely, obtaining SVGs by testing the variance components associated with overall spatial effects and obtaining ctSVGs by testing the coefficients associated with cell types. As a computationally scalable framework, NCTDA enables robust research of large-scale spatial transcriptomics data. Through simulation and real data applications, the results confirm the accuracy and efficiency of NCTDA, enabling it to characterize distinct cellular states and gene modules within structurally complex tissues comprising multiple cell types. It delivers more profound insights into the spatial expression characteristics of genes and cellular functional heterogeneity during tissue development processes and disease states.