Assessing Machine Learning Model Stability and the Value of Hyperparameter Transfer: Effects of Sample Size and Dataset Heterogeneity

This study investigates two critical factors affecting the performance of traditional machine learning (ML) models: training dataset size and the impact of integrating external, potentially noisy data (dataset heterogeneity). Additionally, we evaluate the effectiveness of hyperparameter transfer. Experiments utilized three distinct COVID-19 blood test datasets: the Wuhan dataset (356 cases), used as the source for hyperparameter extraction; the San Raffaele Hospital (OSR) dataset (1,624 cases), serving as the primary target for training, testing, and creating balanced subsets of varying sizes; and the Istituto Ortopedico Galeazzi (IOG) dataset (58 cases), which was integrated to simulate heterogeneity due to its collection via different devices and clinical settings. The discussion extends to a simple TL approach in which optimal hyperparameters were transferred from the Wuhan source to OSR target models. Both Gradient Boosting Machine (GBM) and Generalized Linear Model (GLM) algorithms were tested in traditional and TL settings. Performance was evaluated using accuracy, area under the curve, sensitivity, and specificity. Results show that data quantity and quality substantially influence model performance, with saturation typically observed around 450–500 observations. The addition of external, noisy data (IOG) significantly impacted specificity, particularly in GBM models trained on smaller OSR samples. Specifically, the GBM-based hyperparameter transfer learning (HTL) model demonstrated a “crossover” effect; while it initially lagged behind traditional ML in accuracy at low data volumes (0.74 vs. 0.78 at n = 50), it surpassed ML in larger datasets, reaching a peak accuracy of 0.87 (vs. 0.84 for ML) at n = 1,100. However, due to heterogeneity, this approach also exhibited sensitivity fluctuations in high-data regimes. GLM models demonstrated greater robustness with consistent accuracy levels (~0.84) and minimal variance, possibly due to their simpler structure and better alignment with underlying data characteristics. This study highlights the importance of considering both training data volume and integrity in ML and TL applications. While increasing the quantity of data can enhance performance, the unfiltered inclusion of noisy external data may degrade it. Hyperparameter transfer is a promising strategy for maintaining model reliability across varying data conditions.