Elements of Data Quality: A Foundation for Data Trust (V1)

The abundance of new data sources has created massive opportunities to transform society. For those opportunities to be realized, users must know what data are good for what purpose, and what data are not. What data are of sufficient quality to be trusted? Scientists and legislators, and other users like the public at large, often don’t know. The sheer volume and heterogeneity of data, as well as the variety of different ways in which data can be used, is overwhelming. The consequences of the lack of public signals of data quality are becoming evident. The potential value of new technologies like Artificial Intelligence is threatened: models built on low quality data too often “hallucinate” results. Businesses that have access to high quality data have a substantial advantage over those that do not. And highly publicized cases of the lack of data quality, and even data fraud, have shaken public trust in science. The focus of this paper is to describe an approach to measuring the elements of data quality in a way that can produce understandable signals to the user community. Developing such signals is operationally difficult because of (at least) three challenges. One is that, although much progress has been made in terms of identifying the elements of production data quality that progress has not been translated into a scalable set of signals, or rubric. Another is that data quality is not a monolithic attribute of a particular dataset, but rather a “context-dependent attribute that must be assessed relative to specific goals and users” . A third is that the importance of community engagement to define the use to which the data will be put, contribute to the knowledge base on which models can be built, and build trust in the data. This last is particularly important, since trust is not a static construct, but evolves over time based on a series of observations and interaction.