Interpretable LightGBM Framework for Predicting Esophageal Cancer Using Clinical Data: Integrating Ensemble Feature Selection and Data Balancing

Esophageal cancer is a very aggressive cancer with low chances of survival mainly because of late diagnosis and complexity in clinical heterogeneity. In spite of promising results in terms of early cancer prediction, most current studies have limitations due to the lack of classes balance, risk of data leaking, inadequate model comparison, and limited interpretability. To overcome such problems, this paper suggested a strong and explainable machine learning architecture to predict esophageal cancer using structured clinical data. The proposed methodology uses extensive preprocessing, such as feature elimination based on leakage, missing values, feature encoding and scaling, and an ensemble feature selection algorithm, which works on the Chi-square and Mutual Information methods. The systematic assessment of model performance was conducted in three experimental conditions, namely, without data balancing, SMOTE, and IHT. Various machine learning and deep learning models were evaluated by various evaluation metrics to make sure that they are robust and generalized. The experimental findings prove that data balancing considerably enhances predictive performance. LightGBM trained on SMOTE-balanced data performed the best out of all the considered models, with an accuracy of 99.86% and the highest ROC-AUC score. SHAP and LIME were used to offer global and local explanations to increase model transparency, which showed features that were clinically meaningful. The proposed framework provides a reliable, interpretable, and effective decision-support system, which points to its possible practice in clinical settings when it comes to esophageal cancer prediction.