A systematic review of user-centered evaluations of exploratory data visualization tools

Exploratory data visualization tools help users visually build a better understanding from a dataset they do not fully know yet. To ensure tools are useful, authors often conduct user evaluations. However, practices are not formalized nor standardized, and every author makes their own protocol. This paper aims to explore data visualization authors’ practices in evaluating their tools. We conducted a systematic literature review and, when relevant, reported it with the guidance of the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) Statement. In January 2025, we sought 32 recent and influential dataviz papers from Google Scholar and Semantic Scholar. These reports are about exploratory data visualization; they all present a new tool and include a user evaluation. Drawing from the user-centered design field, we identified 56 dimensions about those reports and synthesized our findings by categorizing each aspect of the protocols (main protocols, data they produced, participant management). From these insights, we discussed best practices, open science, and biases (social desirability bias, non-probability sampling, Hawthorne effect). This study provides an overview and a categorization of user-centered testing notions. It provides general knowledge to help authors form informed opinions when building user-centered protocols for data visualization. However, this study does not provide comprehensive guidelines in how to create protocols, as we are not aware of any empirical data that helps compare protocols to one another.