Diagnostic classification of hepatocellular carcinoma in the Vietnamese population using clinical and biochemical data: explainable machine learning approaches

Hepatocellular carcinoma (HCC) is one of the leading causes of cancer-related deaths worldwide and one of the leading causes of cancer-related death in Vietnam. This highlights the need for accurate and interpretable diagnostic classification models based on routine clinical parameters. This study investigates machine learning (ML) techniques for HCC classification and explanation using clinical and biochemical data, particularly in the context of imbalanced ratio between HCC and non-HCC. A transparent ML framework was developed, focusing on feature selection and model interpretability. First, the VD_HCC dataset, comprising 880 Vietnamese patients (303 with biopsy-confirmed HCC and 577 non-HCC), was collected at a Vietnamese healthcare center. A pre-processing pipeline was then applied to ensure data consistency, reduce noise, and improve downstream model performance. Next, eight feature selection methods were used to identify important biomarkers. Features consistently selected across multiple methods were combined with clinical expertise to build optimized feature subsets. Several classifiers, including ensemble methods such as XGBoost, AdaBoost, and LightGBM, were evaluated using these feature sets. A two-layer XAI analysis combining SHAP, LIME, Partial Dependence Plots (PDPs), and a feature-correlation graph was then used to compare feature attributions and characterize fitted-model behavior and associations in the study data. Across 30 repeated runs with independent training and test partitions, XGBoost using the 10 features set (named Set 5) achieved an Accuracy of \(0.938 ± 0.018\) and an F1-score of \(0.909 ± 0.022\) , with AFP, HBsAg, CEA, AST, Platelets, and ALT among the leading contributors. On the locked hold-out test set, both XGBoost and LightGBM achieved the best ROC-AUC values of 0.979 and PR-AUC values of 0.971 and 0.965, respectively. Under the most imbalanced stress-test scenario (HCC = 1:21), XGBoost and LightGBM achieved comparable Accuracy; XGBoost showed higher Precision, Recall, F1-score, and ROC-AUC, whereas LightGBM achieved a slightly higher PR-AUC ( \(0.878 ± 0.089\) vs. \(0.861 ± 0.093\) ), compared with a no-skill PR-AUC baseline of 0.045. For the final Set 5 XGBoost model, aggregated LIME and SHAP yielded highly consistent feature rankings Spearman \(ρ = 0.976\) ; top-six Jaccard index = 1.000, while the feature-correlation graph provided a complementary description of associations among the study variables. These explanation analyses characterize the behavior of the fitted model and associations within the study data, rather than establishing causal effects or independently validated clinical importance. The ensemble classifiers showed relatively stable discriminative performance across imbalance levels, despite heavy HCC imbalance on the test set. The proposed framework provides an interpretable view of HCC classification by identifying the features that contribute most to the model’s predictions in separating HCC from non-HCC cases.