Scalable Poisson regression via divide and recombine with asymptotic theory, Monte Carlo simulation, and application to cardiovascular risk factor burden in BRFSS

Large health surveillance datasets can make standard Poisson regression computationally expensive, even on well-equipped workstations. We develop and evaluate a divide-and-recombine framework for Poisson regression combining computational scalability with formal inferential guarantees. We establish consistency, asymptotic normality, and efficiency of the aggregated estimator, with an explicit O ( S / n ) efficiency-loss bound as subsets increase. Finite-sample performance is evaluated in 1,000 Monte Carlo replications with sample sizes from 100,000 to 5,000,000. We apply it to a pooled Behavioral Risk Factor Surveillance System (BRFSS) dataset of 2,398,073 observations (2015–2024) for cardiovascular risk factor burden, and benchmark against glm() and speedglm() on a 5-million-row synthetic dataset. Across simulations the method showed negligible bias and near-nominal 95% confidence interval coverage. A low-count study (mean counts 1–5, correlated predictors) likewise preserved unbiased estimation and coverage. Further experiments confirmed robustness to serial dependence, compatibility with survey weights, parity with a communication-efficient distributed estimator, and one to two orders of magnitude greater efficiency than single-subset data reduction. Its principal computational advantage is memory: peak working memory fell by up to 98% versus the full-data fit, with reductions sustained across all sample sizes and count levels. Computation time fell 57–58% in the Monte Carlo study; a low-dimensional low-count study confirmed the same direction of effect at smaller scale, with D&R slower than the full-data fit only at \(n = 10^5\) and 13–28% faster by \(n = 10^6\) . In the BRFSS application coefficients differed from the full-data fit by at most 0.003, with 43–55% lower time; in the synthetic benchmark time fell 62% versus glm() and 19% versus speedglm() . Divide-and-recombine Poisson regression closely reproduces full-data likelihood-based inference while sharply reducing memory and storage on commodity hardware, making it practical for large-scale surveillance.