Abstract A common peculiarity of large-scale applied studies is that the relevant features (signals) are sparse and it is of interests to shrink down the focus toward a much smaller subset in a systematic way. This line of inquiry is of special interest for parsimonious modeling in high-dimensional settings where the goal is to balance model complexity, interpretability, and predictive accuracy. Motivated by microbiome and metagenomic research where high dimensionality is combined with an inherently compositional nature of the data, we formulate an innovative, data-driven framework for selection of sparse, signal-bearing segments ( subcompositions ) hidden in a long sequence of compositional data. Operating with subcompositions as analysis units, the proposed framework is furnished with a broad class of integral probability metrics (IPMs) to quantify a subcomposition’s signal strength, which in turn reflects its influence on the association between the subcomposition’s microbial community and a host phenotype, or any other health/disease-related factor of interest. This class of IPMs is general enough to allow for the incorporation of structural information underlying the data, such as phylogenetic relationships among microbial communities. At the heart of our framework is a threshold-based Selector of Influential Subcompositions (), an algorithm which works as a screen-and-clean scheme outputting the set of influential subcompositions: the data is subject to screening based on empirical signal strength, and threshold cleaning to filter out the subcompositions that contain no signals. sets the selection threshold in a data-driven fashion by adapting the recent notion of CsCsHM statistic, proposed as an optimal signal detection procedure in a sparse, two-component mixture model. We validate the proposed framework through extensive numerical experiments, including challenging cases with sparse, high-dimensional real-world datasets, and demonstrate its effectiveness in identifying clinically relevant influential subcompositions for differentiation of the vaginal microbiome taxonomic profiles in a cohort of Swedish women. The code is available at https://github.com/annti71/Selins .
Pavlenko et al. (Sat,) studied this question.