A Hybrid Machine Learning–Logistic Regression Pipeline for Risk Factor Identification in High-dimensional Epidemiological Data with Extremely Low Events Per Variable

Abstract Background In epidemiological cohort studies with rare binary outcomes and high-dimensional predictors, traditional univariate screening often fails after multiple-testing correction, halting analysis. Low events per variable (EPV) ratios exacerbate instability in multivariable models. We propose a hybrid pipeline integrating machine learning (ML) feature selection with logistic regression to identify risk factors via odds ratios (ORs) in such settings. Methods We applied the pipeline to a dataset with 1,265 observations, 138 predictors, and ~ 50 positive events (EPV = 0.36). Univariate tests (Fisher’s exact/χ²) with false discovery rate (FDR) correction failed. ML-based feature selection (tree-based importance) was used to identify key predictors, which are then fit in multivariable logistic regression on a high-performance computing (HPC) system to estimate odds ratios (ORs). Results Univariate screening yielded no significant predictors post-FDR correction. ML selected 80 features, with four showing significant ORs in logistic regression, demonstrating the pipeline’s ability to uncover associations missed by traditional methods. Conclusions This pipeline offers a reproducible framework for epidemiological studies with rare outcomes and high-dimensional data, balancing computational feasibility and interpretability.