The Meta-Meta-Analysis: A Systematic Map of What 200 AI Benchmark Studies Actually Measured

Key Points

• To systematically map what AI benchmark studies have measured.
• Analyzed 200 AI benchmark studies
• Created a systematic map of measurements used
• Classified studies based on key metrics
• Identified diverse measurement approaches across AI benchmarks
• Highlighted inconsistencies in evaluation metrics