Understanding the decision-making behavior of machine learning models is essential in domains where individual predictions matter, such as medical diagnosis or sports analytics. While explainable artificial intelligence (XAI) methods such as SHAP provide instance-level feature attributions, they mainly summarize typical decision behavior and offer limited support for systematically exploring atypical yet correctly classified cases. In this work, we introduce the Classification Outlier Variability Score (COVAS), a framework designed to support hypothesis generation through the analysis of explanation variability. COVAS operates in the explanation space and builds directly on SHAP value representations. It quantifies how strongly an individual instance’s SHAP-based explanation deviates from class-specific attribution patterns by aggregating standardized SHAP deviations into a single score. Consequently, the applicability of COVAS inherits the model- and data-agnostic properties of SHAP, provided that explanations can be computed for the underlying model and data. We evaluate COVAS on publicly available datasets from the medical and sports domains. The results show that COVAS reveals explanation-space outliers not captured by feature-space outlier detection or prediction uncertainty measures. Robustness analyses demonstrate stability across parameter choices, class imbalance, model initialization, and model classes. Overall, COVAS complements existing XAI techniques by enabling targeted instance-level inspection and facilitating XAI-guided hypothesis formulation.
Roth et al. (Mon,) studied this question.