Benchmarking: Past, Present and Future

Where have we been, and where are we going? It is easier to talk about the past than the future. These days, benchmarks evolve more bottom up (such as papers with code). 1 There used to be more top-down leadership from government (and industry, in the case of systems, with benchmarks such as SPEC). 2 Going forward, there may be more top-down leadership from organizations like MLPerf 3 and/or influencers like David Ferrucci 4 . Tasks such as reading comprehension become even more interesting as we move beyond English. Multilinguality introduces many challenges, and even more opportunities.