Clustering ensemble method integrating Gaussian mixture model and three-way decision (GMM-3WD-CE)

Clustering ensemble improves clustering quality by integrating multiple base clustering results; however, existing methods suffer from inadequate handling of boundary uncertainty and lack a unified probabilistic-to-decision framework. This paper proposes GMM-3WD-CE, which integrates Gaussian Mixture Model (GMM) with three-way decision (3WD) theory to construct a multi-level uncertainty modelling framework. The method generates $$M=50$$ diverse base clusterings via a multi-algorithm strategy, constructs a weighted co-association matrix using quality scores derived from the silhouette coefficient, the Caliński–Harabasz index, and the Davies–Bouldin index, employs the ICL criterion for optimal GMM model selection, and adaptively calculates three-way decision thresholds through the Otsu algorithm to partition samples into core, boundary, and trivial domains. Differentiated label-assignment strategies for each region yield the final consensus clustering. Comparative experiments on eight benchmark datasets with nine comparison methods show that GMM-3WD-CE achieves statistically significant average improvements of $$3.2%$$ in NMI and $$3.9%$$ in ARI over PCPA and $$8.8%$$ in NMI and $$10.4%$$ in ARI over classical MCLA, while remaining competitive with the strongest recent baseline, SDGCA ($$1.2%$$ average NMI advantage; Wilcoxon $$p=0.089$$, medium effect size $$d=0.41$$). Ablation experiments verify the contribution of each component; Wilcoxon and Friedman tests with Cohen’s d effect sizes confirm statistical significance against all other baselines; and runtime/scalability analyses characterise the computational trade-offs.