On feature selection to disentangle cell type and state transcriptional programs

Unlabelled: Single-cell omics approaches profile molecular constituents of individual cells. Replicated multi-condition experiments in particular aim at studying how the molecular makeup and composition of cell subpopulations changes at the sample-level. Two main approaches have been proposed for these tasks: firstly, cluster-based methods that group cells into (non-overlapping) subpopulations based on their molecular profiles and, secondly, cluster-free but neighborhood-based methods that identify (overlapping) groups of cells in consideration of cross-condition changes. In either approach, discrete cell groups are subjected to differential testing across conditions; and, a low-dimensional cell embedding, which is in turn derived from a subset of selected features, is required to delineate subpopulations or neighborhoods. We hypothesized that decoupling differences in cell type (i.e., between subpopulations) and cell state (i.e., between conditions) for feature selection would yield an embedding space that captures different aspects of cellular heterogeneity. And, that type-not-state embeddings would arrive at differential testing results that are more comparable between cluster- and neighborhood-based differential testing approaches. Our study leverages a simulation framework with competing type and state effects, as well as an experimental dataset, to evaluate a set of feature scoring and selection strategies, and to compare results from downstream differential analyses. Supplementary information: The online version contains supplementary material available at 10.1186/s12864-025-12085-9.