Effective Predictive Modelling for Emergency Department Visits and Evaluating Exogenous Variables Impact: Using Explainable Meta-learning Gradient Boosting

Accurately predicting Emergency Department (ED) visits is essential for optimising resource allocation, including staffing adjustments and Operating Room scheduling. Despite the proliferation of AI-driven models, effective ED visit prediction remains challenging due to limited generalisability, susceptibility to overfitting and underfitting, scalability, and the complexity of fine-tuning hyper-parameters. To address these challenges, we propose a novel Meta-learning Gradient Booster (Meta-ED) approach to forecast daily ED visits. Meta-ED leverages a comprehensive dataset spanning 23 years from Canberra Hospital, incorporating exogenous variables such as socio-demographic characteristics, healthcare usage, chronic diseases, diagnoses, and climate parameters. Meta-ED combines four foundational learners—CatBoost, Random Forest, Extra Trees, and LightGBM—with a Multi-Layer Perceptron (MLP) as the master-level learner, thereby enhancing predictive precision by integrating the strengths of diverse base models. Our comparative analysis, which involved testing 23 models against a set of predefined criteria, demonstrates the superior performance of Meta-ED, achieving an accuracy of 85.7% (95% CI [85.4%, 86.0%]) and outperforming prominent models like XGBoost, Random Forest, AdaBoost, LightGBM, and Extra Trees by up to 106.3%. Furthermore, incorporating climate features resulted in a 3.25% improvement in prediction accuracy, effectively capturing seasonal variations that influence patient volumes. These results underscore the potential of Meta-ED to advance predictive analytics in complex healthcare environments.