An Explainable and Optimized Ensemble Learning Framework Using XGBoost, CatBoost, and LightGBM for Predictive Analytics

AbstractPredictive analytics on complex tabular datasets is a cornerstone of data-driven decision-making across high-stakes domains including financial risk assessment, healthcare prognostics, and customer intelligence. Although Gradient Boosted Decision Tree (GBDT) architectures—predominantly XGBoost, CatBoost, and LightGBM—exhibit superior empirical predictive power over deep neural networks on tabular structures, three foundational challenges persist: (1) high sensitivity to complex, non-convex hyperparameter spaces; (2) vulnerability to data leakage and validation bias; and (3) a notorious "black-box" opacity that impedes regulatory compliance and clinical adoption. This study proposes an explainable, Bayesian-optimized stacking ensemble learning framework that addresses these challenges systematically. The framework integrates XGBoost, CatBoost, and LightGBM as heterogeneous base-level learners, rigorously tuned via Tree-Structured Parzen Estimator (TPE) Bayesian optimization within an Optuna framework across 50 trials per algorithm under leakage-free nested 5-fold stratified cross-validation. Out-of-fold (OOF) predicted probability vectors are fed into an ElasticNet regularized logistic meta-learner to synthesize orthogonal decision boundaries. Experimental results on an empirical benchmark (N = 30,000) demonstrate that the proposed Stacking Meta-Ensemble outperforms all individual models (ROC-AUC: 0.7891, p < 0.001) while delivering full post-hoc interpretability via TreeSHAP. Underlying Benchmark DatasetThe underlying dataset utilized in this paper is deposited on Zenodo with DOI: https://doi.org/10.5281/zenodo.23667925