A Stability‐Enhanced Lasso Approach for Covariate Selection in Non‐Linear Mixed Effects Model

ABSTRACT Non‐linear mixed effects models (NLMEMs) defined by ordinary differential equations (ODEs) are central to modeling complex biological systems over time, particularly in pharmacometrics, viral dynamics, and immunology. However, selecting relevant covariates associated with dynamics in high‐dimensional settings remains a major challenge. This study introduces a novel model‐building approach called Lasso‐SAMBA (SAMBA: Stochastic Approximation for Model Building Algorithm) that integrates Lasso regression with a stability selection algorithm for robust covariate selection within ODE‐based NLMEMs. The method iteratively constructs models by coupling penalized regression with mechanistic model estimation using the SAEM algorithm. It extends a prior strategy named SAMBA, originally based on stepwise inclusion, by replacing this step with a penalized, stability‐driven approach that reduces false discoveries and improves selection robustness. By maintaining the monotonic decrease of the information criterion through a calibrated exploration of penalization parameters, the proposed method outperforms conventional stepwise and Bayesian variable selection alternatives. Extensive simulation studies, spanning pharmacokinetic and immunological models, demonstrate the superiority of Lasso‐SAMBA in variable selection fidelity, FDR (False Discovery Proportion) control, and computational efficiency. The Lasso‐SAMBA method is implemented in an R package. Applied to a Varicella‐Zoster virus vaccination study, the method reveals robust, biologically plausible associations between parameters of the mechanistic model of the humoral immune response and early transcriptomic expressions. These results underscore the practical utility of our method for high‐dimensional model building in systems vaccinology and beyond.