rHMM: A New Regularized Algorithm-based Hidden Markov Model for High-dimensional Longitudinal Data Analysis

Abstract Many real-world underlying processes, such as addiction status and its longitudinal change associated with substance use, which are influenced by multifaceted factors, are not directly observable but can be captured by competent statistical methods. Hidden Markov models (HMMs) are a powerful tool for characterizing temporal dependence and mixture of latent variables in longitudinal data analysis , whereas the current HMM approach is largely limited in high-dimensional settings due to lack of dimension reduction and variable selection techniques. This article presents a novel approach tailored to address these limitations within the framework of HMMs. To reduce bias in parameter estimation by effectively discerning and excluding insignificant covariates, we propose a two-step algorithm that integrates HMMs with regularized generalized linear models (GLMs). Initially , we combine the coordinate descent algorithm with the EM algorithm to identify significant covariates. Subsequently, we recalibrate the HMM using the selected covariates employing maximum likelihood estimation for precise parameter refinement. Simulations demonstrate our method can accurately identify significant covariates and provide unbiased estimates for both the Markov transition and GLM parameters. The proposed method has improved performance over regularized GLMs, with a false negative rate up to 38.2% lower in variable selection. In two real data analyses, our method yields meaningful results that estimate the level of persistence of the unobserved states and are consistent with the relevant studies. Additionally, it effectively selects the set of important 1 covariates. Taken together, the regularized algorithm-based HMM outperforms existing methods, offering a valuable addition to the analytical armamentarium.