Comparison of Internal Validation Strategies for Estimating Machine Learning Model Performance in One-Year Mortality Prediction in Hepatocellular Carcinoma

Background: Machine learning (ML) performance estimates in hepatocellular carcinoma (HCC) may depend on internal validation strategy, especially in small datasets. Aim: To develop models for one-year mortality prediction and compare performance, calibration, decision-analytic performance, and uncertainty estimates using a conventional train–test split and nested cross-validation (nCV). Methods: We analyzed a public dataset of 165 patients with HCC, including 63 one-year deaths. Random Forest (RF), XGBoost, and CatBoost were developed using fold-specific preprocessing, cost-sensitive learning, Optuna-based hyperparameter optimization, and training-based threshold selection. L2-penalized logistic regression provided a conventional benchmark. Models were evaluated using a stratified 80/20 split and five-fold nCV with five inner folds. Six stratified splits assessed partition variability, and unweighted models assessed the effect of class weighting. Discrimination, threshold-dependent classification, calibration, and decision-analytic performance were assessed using ROC-AUC, PR-AUC, balanced accuracy, Matthew’s correlation coefficient, Brier score, calibration curves, and decision-curve analysis. Confidence intervals used 2000 bootstrap resamples; SHAP was used for interpretation. Results: In the conventional split, RF and CatBoost achieved the highest test-set ROC-AUC (0.892 each), while RF achieved the highest PR-AUC (0.860). Under nCV, RF had the highest mean outer-fold ROC-AUC (0.788) and pooled out-of-fold ROC-AUC (0.784). XGBoost had the highest mean PR-AUC (0.673), whereas CatBoost had the highest pooled PR-AUC (0.648). RF had the lowest mean Brier score (0.188), followed by XGBoost (0.194) and CatBoost (0.203). Calibration varied across models, with pooled calibration slopes of 1.438, 0.691, and 0.427, respectively. Across six 80/20 splits, mean ROC-AUC values were 0.792, 0.769, and 0.761 for RF, XGBoost, and CatBoost, respectively, with substantial partition-dependent variation. Removing class weighting yielded broadly comparable performance. Bootstrap confidence intervals were narrower for pooled nCV predictions, but the evaluation samples differed in size (165 versus 33 patients), precluding attribution of this difference to validation strategy alone. Decision-curve analysis was exploratory because no clinically actionable threshold was prespecified. SHAP identified AFP, ALP, hemoglobin, albumin, performance status, and AST among the most influential RF predictors; the highest-ranked predictors were generally consistent across outer folds. Conclusions: nCV separated model development from evaluation and reduced reliance on a single small test set. RF was selected by the prespecified mean outer-fold ROC-AUC criterion, but differences among tree-based models were small and no clear discrimination advantage over penalized logistic regression was demonstrated. External validation is needed before clinical use.