Quantified Reproducibility Assessment for Four Common Types of Evaluation Results in NLP/ML

Abstract Reproductions in NLP/ML indicate worryingly low levels of reproducibility, but their results are hard to interpret, compare, and learn from, because reported in qualitative terms with varying, often unstated, criteria. We propose QRA, a quantitative approach to reproducibility assessment that (i) produces continuous-valued degree of reproducibility assessments at three levels of granularity; (ii) utilises reproducibility measures that are comparable across different studies; and (iii) grounds expectations about degree of reproducibility in similarity between experiments. QRA facilitates more informative reproducibility assessments and conclusions about the causes of better/poorer reproducibility. We demonstrate its benefits by applying QRA to three sets of comparable experiments, obtaining clear evidence that degree of reproducibility depends on similarity of experiment properties, system type and evaluation method.