Addressing Data Scarcity and Class Imbalance Using Active Learning and Synthetic Data Generation: Experimentation on Clinical Datasets

The rapid advancement of Machine Learning (ML) technologies has revolutionized healthcare, particularly in the domains of disease prediction and diagnosis. Nevertheless, some data-related issues, such as annotated data scarcity, low-quality labels, and class imbalance in the dataset, can compromise the effectiveness of the learning process. This paper first gives an overview of Active Learning (AL), which is a strategy in machine learning that helps models to achieve better performance using fewer labeled examples. It then reviews various strategies for addressing dataset imbalance, in particular focusing on data-level rebalancing approaches. The paper provides a review of existing works on active learning in the presence of class imbalance and on the integration of active learning with synthetic oversampling techniques. The central core of the paper is the implementation of four different strategies for data selection from publicly available clinical datasets: Random, Random + SMOTE, Uncertainty, and Uncertainty + SMOTE. We calculated and plotted balanced accuracy and average precision by iterating over 20 different seed values, using cumulative batches of labeled data, for each combination of dataset and classifier. Moreover we also carried out a statistical test to assess the extent to which the difference between the Random and Uncertainty distributions, and that between the Uncertainty and Uncertainty + SMOTE distributions, was statistically significant. Experiments show that uncertainty-based strategy can improve predictive performance compared with random acquisition, particularly with Random Forest, although the observed benefits vary across datasets, classifiers, performance metrics, and annotation budgets. The additional contribution of SMOTE is also context dependent and does not yield statistically significant improvements over uncertainty-based acquisition alone. For the WBCD dataset, it can be seen that Uncertainty + SMOTE is comparable to, or in some cases below, the uncertainty-only strategy. Metrics such as precision, recall, specificity, F1-score, AUROC and AUPRC are evaluated and provided.