Devising domain- and model-agnostic evaluation metrics for generative models an important and as yet unresolved problem. Most existing metrics, which tailored solely to the image synthesis setup, exhibit a limited capacity diagnosing the different modes of failure of generative models across application domains. In this paper, we introduce a 3-dimensional metric, (\α-Precision, \β-Recall, Authenticity), that the fidelity, diversity and generalization performance of any model in a domain-agnostic fashion. Our metric unifies statistical measures with precision-recall analysis, enabling sample- and-level diagnoses of model fidelity and diversity. We introduce as an additional, independent dimension (to the-diversity trade-off) that quantifies the extent to which a model training data -- a crucial performance indicator when modeling sensitive with requirements on privacy. The three metric components correspond to(interpretable) probabilistic quantities, and are estimated via sample-level classification. The sample-level nature of our metric inspires a novel case which we call model auditing, wherein we judge the quality of samples generated by a (black-box) model, discarding low-quality and hence improving the overall model performance in a post-hoc manner.
No takes yet. Share an insight, caveat, or question.
Alaa et al. (2021) studied this question.