Why Comparing Single Performance Scores Does Not Allow to Draw Conclusions About Machine Learning Approach

Developing state-of-the-art approaches for specific tasks is a major driving in our research community. Depending on the prestige of the task, it can come along with a lot of visibility. The question arises how are our evaluation methodologies to compare approaches? One common methodology to identify the state-of-the-art is to partition data a train, a development and a test set. Researchers can train and tune approach on some part of the dataset and then select the model that best on the development set for a final evaluation on unseen test data. scores from different approaches are compared, and performance differences tested for statistical significance. In this publication, we show that there is a high risk that a statistical in this type of evaluation is not due to a superior learning. Instead, there is a high risk that the difference is due to chance. example for the CoNLL 2003 NER dataset we observed in up to 26% of the type I errors (false positives) with a threshold of p < 0. 05, i. e. , concluding a statistically significant difference between two identical. We prove that this evaluation setup is unsuitable to compare learning. We formalize alternative evaluation setups based on score.