Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research

Benchmark datasets play a central role in the organization of machine research. They coordinate researchers around shared research problems serve as a measure of progress towards shared goals. Despite the role of benchmarking practices in this field, relatively little has been paid to the dynamics of benchmark dataset use and reuse, or across machine learning subcommunities. In this paper, we dig into dynamics. We study how dataset usage patterns differ across machine subcommunities and across time from 2015-2020. We find increasing on fewer and fewer datasets within task communities, significant of datasets from other tasks, and concentration across the field on that have been introduced by researchers situated within a small of elite institutions. Our results have implications for scientific, AI ethics, and equity/access within the field.