MCMC Methods: From Theory to Distributed Hamiltonian Monte Carlo over PySpark

The Hamiltonian Monte Carlo (HMC) method is effective for Bayesian inference but suffers from synchronization overhead in distributed settings. We propose two variants: a distributed HMC (DHMC) baseline with synchronized, globally exact gradient evaluations and a communication-avoiding leapfrog HMC (CALF-HMC) method that interleaves local surrogate micro-steps with a single–global Metropolis–Hastings correction per trajectory. Implemented on Apache Spark/PySpark and evaluated on a large synthetic logistic regression (N=107, d=100, workers J∈{4,8,16,32}), DHMC attained an average acceptance of 0.986, mean ESS of 1200, and wall-clock of 64.1 s per evaluation run, yielding ≈18.7 ESS/s; CALF-HMC achieved an acceptance of 0.942, mean ESS of 5.1, and 14.8 s, i.e., ≈0.34 ESS/s under the tested surrogate configuration. While DHMC delivered higher ESS/s due to robust mixing under conservative integration, CALF-HMC reduced the per-trajectory runtime and exhibited more favorable scaling as inter-worker latency increased. The study contributes (i) a systems-oriented communication cost model for distributed HMC, (ii) an exact, communication-avoiding leapfrog variant, and (iii) practical guidance for ESS/s-optimized tuning on clusters.