Key result
A novel Bayesian test of statistical independence effectively evaluated classifier-based diagnosis on unbalanced datasets, providing a measure of evidence that overcomes the limitations of standard metrics.
A Bayesian test of statistical independence provides a more reliable evaluation of classifier performance on unbalanced diagnostic datasets than standard metrics like prediction accuracy.
May aid evaluation of diagnostic classifiers on imbalanced data; leaves open prospective validation before clinical use.
Machine learning techniques are increasingly adopted in computer-aided diagnosis. Evaluation methods for classification results that are based on the study of one or more metrics can be unable to distinguish between cases in which the classifier is discriminating the classes from cases in which it is not. In the binary setting, such circumstances can be encountered when data are unbalanced with respect to the diagnostic groups. Having more healthy controls than pathological subjects, datasets meant for diagnosis frequently show a certain degree of unbalancedness. In this work, we propose to recast the evaluation of classification results as a test of statistical independence between the predicted and the actual diagnostic groups. We address the problem within the Bayesian hypothesis testing framework. Different from the standard metrics, the proposed method is able to handle unbalanced data and takes into account the size of the available data. We show experimental evidence of the efficacy of the approach both on simulated data and on real data about the diagnosis of the Attention Deficit Hyperactivity Disorder (ADHD).
No takes yet. Share an insight, caveat, or question.
Olivetti et al. (2014) studied Attention Deficit Hyperactivity Disorder (ADHD) (n=923). Bayesian test of statistical independence vs. Standard classification metrics (Accuracy, MCC, F1, kappa, J) was evaluated on Evaluation of classification results (Bayes factor B10). A novel Bayesian test of statistical independence effectively evaluated classifier-based diagnosis on unbalanced datasets, providing a measure of evidence that overcomes the limitations of standard metrics.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: