Decentralized EM algorithm for Gaussian mixtures under data heterogeneity and partial labeling

Abstract We systematically study several network-based Expectation–Maximization (EM) algorithms for the Gaussian mixture model within decentralized federated learning (DFL). Our theoretical investigation reveals that directly extending the classic EM algorithm to DFL leads to a seriously biased estimator if the data are heterogeneously distributed across different sites. To address this issue, we introduce a momentum network EM (MNEM) algorithm, which integrates information from both current and historical estimators from previous DFL iterations. We further develop a semi-supervised MNEM (semi-MNEM) algorithm, which utilizes valuable information provided by partially labeled data. Rigorous theoretical analysis demonstrates that the MNEM estimator can achieve the same asymptotic efficiency as the whole sample estimator under appropriate regularity conditions, even if the data are heterogeneously distributed. Moreover, the semi-MNEM estimator significantly improves the convergence speed of the MNEM algorithm, even if different mixture components are poorly separated. Extensive simulations are conducted, and a widely used chest X-ray dataset is analyzed to demonstrate the finite-sample performance of the proposed methods.