Penalized Estimation in Finite Mixtures of Multivariate Regression Models via the EM‐PGM Algorithm

ABSTRACT In many areas of medical and life sciences research, multiple diagnostic criteria and phenotypic outcomes are assessed simultaneously to characterize diseases and biological traits. These multivariate outcomes are often associated with high‐dimensional covariates and subject to underlying population heterogeneity, presenting challenges for analysis and interpretation. To address these issues, we study a finite mixture of multivariate regression models (mvFMR), which jointly models multiple outcomes while capturing latent subgroup structure. By accounting for both outcome multiplicity and data heterogeneity, mvFMR improves predictive performance and enhances interpretability for multivariate outcomes in complex datasets. We adopt a penalized maximum likelihood approach to estimate mvFMR, which enables scalability and variable selection in high‐dimensional settings. We further develop the EM‐PGM algorithm, an efficient estimation procedure that combines the expectation‐maximization (EM) framework with the proximal gradient method (PGM) to handle high‐dimensionality and the non‐differentiability of the penalty function. Simulation studies demonstrate that mvFMR with EM‐PGM outperforms alternative methods in terms of estimation accuracy, sparsity recovery, and computational efficiency. Applications to analyses of diabetes diagnosis data and Cancer Cell Line Encyclopedia data illustrate the practical utility of our penalized mvFMR.