Assessing the validity of quality indicators: keep the context in mind!

In many European countries, governments, third-party payers, patient and consumer organisations and healthcare professionals monitor the quality of health care by means of indicators. On the national and international level, policymakers see indicators as an essential tool to keep track of trends and evaluate the impact of policies. Third-party payers are interested in quality indicators to measure their ‘return-on-investment’, to implement pay-for-performance schemes or selective contracting. For patients, quality indicators are an important source of consumer information to facilitate choice. And finally, professionals use quality indicators to continuously monitor and improve their performance. The development of indicators ideally takes place in interaction between researchers and stakeholders.1 In this interactive process, the international scientific literature serves as input for indicator development. However, indicators that have been developed and tested in a certain setting or country need to be adapted to the local context and the specific objective for which they are applied. Adaption to the local context is necessary for several reasons. One is that the information infrastructure that is used to calculate indicator scores differs between health care systems. For instance, a measure like hospital standardized mortality rates (HSMR) has been used in various countries, such as USA, UK, Canada, Australia, Sweden, France and Japan. However, the introduction of this well-known measure in the Dutch health care system still called for additional research on the model used for calculating the HSMR and its statistical performance.2 Another reason why adaptation is important is that health care systems are different in different regions. For instance, HSMRs should be corrected for regional variation in the organisation and performance of health care facilities adjacent to the hospital.3 How this correction should take place and which adjacent facilities should be taken into account is country-specific. In this issue, Fischer et al.4 explored the validity of readmission rates as a quality indicator in the European setting. According to the authors, hospital readmissions cause high costs, pose a burden to patients and can be computed relatively easily from administrative databases. In USA, hospital readmissions have been studied extensively, and the majority of studies reported that readmissions were valid measures of quality that reflect substandard care received during the previous hospitalisation. Following the American example, European countries too introduced readmission rates as a quality indicator. However, Fischer et al. conclude that the amount of research on the indicator’s validity in the European context is scarce. As the validity in Europe has been studied, findings reveal several methodological problems such as the lack of a uniform definition, the impact of case-mix factors and the questionable reliability of the underlying databases. The study by Fischer et al. once more shows that adapting indicators to the local context and the local information infrastructure is important. However, apart from that, the validation of indicators should take the intended use of the indicator into account. As Fischer et al. put it: ‘[…] the actual purpose of the indicator is important when studying its validity, as each type of use (accountability, improvement, consumer information, pay-for-performance) may place different demands on the degree of validity’. In indicator development, the most important question to be kept in mind is Who wants to know what and why?.1 In continuous quality improvement, it is important to measure indicator scores over time. This calls for different statistical models than those applied for comparing the performance of providers at one specific point in time, for example, to inform consumer choice or to facilitate selective contracting. Apart from that, there is a difference between the use of indicator scores in internal quality improvement and the use of those scores for external accountability, consumer choice or pay-for-performance. As a rule of the thumb, the validity, reliability and comparability of indicator scores used for external purposes should be much more robust than for internal use. Researchers should insist on caution every time policy makers or other stakeholders promote the use of indicators outside the context or purpose for which they have originally been developed. A recent example is the use of patient-reported outcome measures (PROMs) to compare the performance of hospitals and other health care providers. This has been introduced in the English National Health Service (NHS) since 2009 and is currently also promoted by health insurers and patient organisations in the Netherlands. PROMs have been collected in the past as part of clinical trials or in clinical registries. However, the use PROMs as a measure of hospital performance is new. This is an important development, because ultimately, health care quality should be judged by the value that is created for patients and their families. However, there are numerous methodological and statistical pitfalls that call for more research.5 On the European level, this research should be coordinated because there is a lot that we can learn from each another. However, the example of the study by Fischer et al. shows that there is a need for research on the national level. Introducing indicators that have been developed in different contexts and for different purposes is not just a matter of copy and paste.