Considerations when evaluating real‐world data quality in the context of fitness for purpose

With the growing use of large real-world clinical datasets comes increased scrutiny about the quality of real-world data (RWD). The question of whether the data are good enough for decision-making is often raised.1 It is important to understand that quality cannot be assessed by looking at data in isolation: Any evaluation of RWD quality underpinning real-world evidence (RWE) must consider whether that data source has the information to answer a given research question. A way forward in the quality conundrum suggested by Girman et al2 is to establish a framework for evaluating data appropriateness, also known as fitness for purpose, meaning the degree to which the chosen data source aligns with the ability to accurately and reliably address the research question being posed. Here, we offer a simple framework for characterizing the attributes of any RWD source and key aspects of research questions to facilitate the optimal matching of research needs and data sources to achieve meaningful and reliable results. Cohort: Who are the data subjects of interest and what characteristics are needed to identify them? Intervention(s): Are there interventions of interest? Outcome(s) of interest: Are there one or more outcomes of interest? Essential covariates: Are there any important covariates that need to be assessed? Timing Once key components of the research question have been assessed, they must be prioritized. Distinguishing the “must-have” data from the “nice-to-have” is critical since there are many RWD sources available and no one individually is likely to match perfectly to the research needs. Knowing which components of the research study are essential will pair the possible database matches more effectively. While the list above is not exhaustive in defining aspects of quality, it will help characterize both the data available and the question to be answered, and it should provide a breadth of consideration for assessing a data source’s fitness for purpose. Many of the issues raised above may trigger more specific critical queries for any given research question. In some cases, not all the information needed to make a full assessment will be available in easily accessible RWD metrics and so specific questions to data source holders and/or feasibility studies may be needed, especially to assure that there are likely to be enough data subjects of interest. As part of the Sentinel Initiative, the FDA and other organizations are developing detailed recommendations on standards-based approaches to describing data and presenting data quality metrics.4, 5 Here, we offer some straightforward information parameters to guide anyone interested in gauging RWD appropriateness for any research question. The speed of achieving results may also affect the choice of RWD source. If the research question concerns a life or death issue where little evidence is available, then even an imperfect RWD source that can deliver results quickly may be useful to generate intermediate evidence as opposed to waiting potentially years for perfect data to develop. Ideally, the data assessments described here should be easily accessible; however, some information may be proprietary so that, for commercial reasons, not all the desired information can be published. It may be helpful to query the data owner, researchers who have used the data, and/or colleagues and professional peers to obtain gain deeper insight. Finally, it is important to remember that RWD sources and the environment they operate in are dynamic and continually updated, so the assessment of quality metrics will need regulator updating. Moreover, good data quality needs to be accompanied by appropriate study design and analyses. On-going efforts to support the regulatory use of real-world data in the US, Europe, and China will help flesh out the methodological and design issues that will help achieve meaningful and reliable results.6-8 While access to RWE and its use is increasing,9 it is natural for scientists to question the “quality” of the “messy” underpinning RWD, especially for those schooled in the use of randomized clinical trials where researchers have active control of data collection, and budgets to support extensive quality control. Understanding quality is particularly important in the regulatory sphere where decisions based on RWE need to be justified judiciously. Even if all the metrics for evaluating RWD quality were established and agreed-on, such criteria are unlikely to be enough for every scientific research purpose. Most RWD sources will be good fits for some research questions but not others. Data appropriateness needs to be gauged by reviewing the strengths and weaknesses of any dataset under consideration in the context of the research initiative, study design, budget, and time available to assemble relevant information. Data quality is not an absolute metric that can predict utility in isolation. A quality assessment can only be evaluated with full knowledge of the research question and immediacy of the need of information. Only through the optimal pairing of data source and research question parameters can you have confidence in delivering a reliable conclusion. The author(s) have no conflicts of interest to declare.