Prevalence-Insensitive Evaluation of Diagnostic Systems Under Class Imbalance: The Harmonic Mean of Per-Class Sensitivity

In safety-critical diagnosis under class imbalance, widely used global metrics—e.g., accuracy, F-score, and MCC—conflate class-conditional behavior with class priors, often yielding overly optimistic assessments of overall performance, thereby hindering stable comparisons across datasets. We formalize the prevalence sensitivity of these indices and advance a prevalence-insensitive evaluation built solely from class-conditional rates (binary TPR/TNR; multiclass per-class sensitivities). We analyze the arithmetic, geometric, and harmonic means as class-symmetric aggregators and provide a geometric characterization of their isocurves. Empirically, on 18 UCI datasets, accuracy exceeded the harmonic mean by ∼3% on average, but the gap widened on skewed sets (e.g., cardiotocography: Acc=0.896 vs. H=0.803, +11.5%). On a 6756-sample, 35-class tumor dataset, high overall agreement (Acc=0.906, Cohen’s κ=0.899) coexisted with dispersed per-class sensitivities (A=0.795, G=0.743, H=0.658) and moderate MCP (0.564), evidencing weak minority-class performance despite strong accuracy. These results show that reliance on accuracy or κ can overstate performance precisely where it matters most, inflating perceived safety and obscuring rare-class failure modes. Prevalence-insensitive, class-symmetric means—especially the harmonic mean of per-class sensitivity—yield conservative, comparable summaries better aligned with risk-aware evaluation and deployment in critical systems.