International benchmarking. Option or illusion?

Quality improvement is and should be an international endeavor. Frameworks in terms of terminologies and definitions as well as philosophies and methods in terms of designs and principles should be shared internationally. This transfer of knowledge and inspiration is widely successful through the international literature, international societies on quality of care, collaboratives and friendships. Such international efforts have resulted in attempts to develop common and comparable definitions. Recent international collaboration on quality improvement has centered on several international projects targeting the development of indicators and tools for quality. The broadest of these ongoing international projects are the Organization for Economic Cooperation and Development's Health Care Quality Indicators (OECD HCQI) Project and The Nordic Minister Council Project on quality measurement and benchmarking. Major important international quality efforts are also being undertaken as part of the European Union's Simpatie project on patient safety and WHO projects, such as the PATH Project. While these projects have different participating countries and slightly different objects, they are all striving for sound methodologies, and principles and international benchmarking. Many health care systems are striving to reach ‘the international level’ for provided health care, be it in terms of quality, access or cost for care. The questions are, however, what ‘the international level’ is and how it can be defined? How well are international results shared and is it possible to do benchmarking at the international level? The international projects document how difficult this really is. A variety of threats to comparability have been identified when it comes to comparisons of quality at the international level. The experience of the OECD HCQI Project and The Nordic Minister Council Project has shown that even for common indicators (30-day in-hospital mortality for stroke and acute myocardial infarction; 5-year survival rates for breast cancer, colorectal cancer and lung cancer; etc.) it is difficult to yield data that are representative to the international nations as a whole. Often reported indicator data for different countries are related to different years. In some projects, data for some indicators differ between countries from 1991 to 2005 and in consequence this cannot be compared. In several cases, in particular with regionally based samples, countries have stated that the results were generalizable to the nation. With regard to cancer data, even prominent countries are only able to report data from one or few individual regions on basic indicators such as survival rates for breast, cervical or colorectal cancer. For other indicators, countries report data from selected providers or care settings that may not reflect the prevailing patterns of care in a given country. Another severe problem is the differential ability of countries to track patients after hospital admissions. Some countries assign unique patient identifiers, but most countries do not. And some countries are able to track patients after discharge, others do not. When comparing outcomes indicators at the international level, it is the rule rather than the exception that risk adjustment is lacking. To these severe potential selection and information biases, there is also the lack of validity testing and lack of exhaustive and exclusive data specifications that are so important in ensuring that the data are truly comparable. Finally, often indicators for quality benchmarking have been chosen not because they reflect the quality of care at best but because they reflect the data source that is available. In other words, we measure what we can measure and not what we want to measure. It thus seems that modern health care systems have problems with documenting their quality and thereby their value. In consequence of this, international comparisons based on nationwide data should be interpreted with outermost caution.. This is, however, not always the case and international comparisons based on unreliable data can lead to actions in the wrong direction. The question is then, what can be done? The main focus with regard to international benchmarking should be: how we can improve our measures and data quality for international comparisons. In other words, the focus should be to find how we now and in the future can develop shared indicators, common data specifications, shared data collection protocols, common risk adjustment strategies, etc. for international benchmarking. This is not an easy task, but it can be done. It is time to stop pretending that we can compare health care systems on existing data and even prepare league tables from the data. This is why the OECD HCQI Project and The Nordic Minister Council Project have expressly aimed at testing data comparability, mapping the greatest threats to validity and made a commitment to working with national data systems to improve data comparability. It should be a shared vision for modern health care systems that they can document their value and quality. However, much more work is needed across countries for this vision to become a reality. At national and international levels, we need to invest in quality measurement systems and in international collaboration. It seems from the literature on indicators benchmarking that it will get paid in terms of improvements of patient care.