Improving parameter estimation and order selection for beta mixture models

Finite mixtures of beta distributions provide a flexible framework for modelling bounded data, such as proportions or rates, and are widely used in applications in genomics and biostatistics. However, estimation is complicated by challenges such as explosive likelihood and non-identifiability. We develop new methodologies to address these issues. We propose a penalized maximum likelihood estimator that stabilizes estimation, along with a score-adjusted moment-based alternative that improves computational efficiency and serves as an effective initialization strategy for the EM algorithm. For order selection under non-identifiable models, we propose a beta mixture reduction method based on composite transportation divergence, which provides an efficient computational tool for generating lower-order mixtures in order selection workflows. The proposed methods are evaluated through simulation studies and applied to DNA methylation data. All implementations are available in the MixtureInf2.0 R package on GitHub.