Who will bell the cat? A call for a new approach for validating blood pressure measuring devices

Blood pressure (BP) measurement should be the foundation stone on which all decisions in hypertension, be they in practice or in research, are dependent. If BP measurement is inaccurate, it follows that all decisions will be flawed. Yet the history of clinical practice and scientific research is replete with examples of disregard for the accuracy of BP measurement. Recognizing that the commonest measurement in medicine is often inaccurate, clinical scientists and the scientific hypertension organizations have made recommendations over the years to improve the technique of measurement and the accuracy of BP measuring devices. The discipline of validating BP measuring devices has developed from primitive origins with ad-hoc protocols to the latest revision of the European Society of Hypertension International Protocol (ESH International Protocol), which is now available on-line (www.dableducational.org) 1. In this issue of the Journal Turner 2 asks: ‘ Can we trust automatic sphygmomanometer validations?’ The essence of his case is that, as hypertension is an undisputed leading risk for cardiovascular morbidity and mortality, the devices used to measure BP should be accurate. However, he questions the present policy of trusting the quality control of sphygmomanometer validations to peer review of published reports, which is not without its limitations. He argues that patients, whose health may depend on accurate BP measurements, are surely entitled to expect that the quality control of validation of sphygmomanometers is at least as good as the quality control of the scales used in the sale of potatoes by a greengrocer. Is it not logical then to apply external accreditation of laboratories that validate sphygmomanometers as is done for other measurements that are important to society? It is perhaps timely to review the history of BP device validation before addressing the implications of this reasonable proposal. Humble beginnings with ad-hoc protocols When the technique of BP measurement was introduced into clinical medicine during the early years of the twentieth century, the importance of accuracy and the limitations of the technique were well recognized 3. However, the standards called for by the clinicians and scientists who pioneered the technique were relaxed as the twentieth century progressed. During the 1960s and 1970s, individual groups, frustrated by the failure of manufacturers to produce evidence to match their often extravagant claims, began to validate BP measuring systems according to a variety of ad-hoc protocols and so illustrated the need for independent validation of devices 4–6. However well intentioned such protocols may have been, they had the serious disadvantage of not permitting comparison of one device against another because of the differing methodologies of validation 7. International protocols In 1987, the Association for the Advancement of Medical Instrumentation published a standard for electronic and aneroid sphygmomanometers, which included a protocol for the evaluation of the accuracy of devices 8. In 1990, the British Hypertension Society (BHS) published a more detailed protocol devoted solely to the validation of devices in the clinical setting 9 and both protocols were revised in 1993 10,11. These protocols, which differed in detail, had a common objective, namely the standardization of validation procedures to establish minimum standards of accuracy and performance, and to facilitate comparison of one device with another. A large number of BP measuring devices were evaluated according to one or both protocols, but experience soon demonstrated that the conditions demanded for validation were extremely difficult to fulfill because of the large number of individuals that needed to be recruited and the ranges of BP required. These factors made validation studies difficult to perform and very costly, with the result that fewer centers were prepared to undertake them. In 2002, the Working Group on Blood Pressure Monitoring of the ESH published an updated protocol, named the International Protocol 12. The ESH Working Group recognized the urgent imperative to provide a simplified protocol that would not sacrifice the integrity of the earlier protocols. The Working Group had the advantage of being able to examine and analyze the data from 19 validation studies performed according to the earlier protocols at the Blood Pressure Unit in Dublin 13. Critical assessment of this database of evidence permitted rationalization and simplification of validation procedures without losing the merits of the much more complicated earlier protocols. The International Protocol was drafted so as to be applicable to the majority of BP measuring devices on the market. The validation procedure was confined, therefore, to adults over the age of 30 years (as these constitute the majority of individuals with hypertension), and it did not make recommendations for special groups, such as children, pregnant women, and the elderly, or for special circumstances, such as exercise. However, the protocol did not preclude manufacturers of devices from subjecting their products to more rigorous assessment and validation. In 2010, the ESH International Protocol was revised based on the evidence and experience acquired from 104 validation studies conducted using the 2002 protocol 14. A further review assessed the impact of the more stringent criteria of the revised ESH International Protocol 2010 on previous studies that had fulfilled the requirements and passed the original ESH International Protocol 2002. If the devices validated according to the earlier protocol 12 were to be validated according to the 2010 revision of the ESH International Protocol 14, the failure rate would increase markedly from 17 to 42%, showing that the revised protocol rightly demands more stringent accuracy criteria in keeping with technological advances Data analyzed by the authors and submitted for publication.. Communicating the results of validation studies For the consumer, whether medical or lay, device accuracy should be of prime importance in selecting a BP measuring device. However, the majority of devices available have not been evaluated independently for accuracy and the consumer often does not have the expertise or information to make a fully informed decision as to which device to purchase. Surveys of the accuracy of BP measuring devices have been reported in the literature from time to time 15–18, but they have the disadvantage of being obtainable only by journal subscribers, and the information soon becomes outdated in that this forum cannot keep pace with the availability of new devices 19. It is generally recognized that it is extremely difficult for doctors and others wishing to purchase BP measuring devices to obtain up-to-date information on the validation status and accuracy of these devices. Device manufacturers complain about the long and costly time lag between validation of a device and the subsequent publication giving it credibility in the marketplace. To overcome these shortcomings a not-for profit trust – the dabl Educational Trust – was established to recruit the necessary expertise to independently evaluate validation studies and claims by manufacturers for their devices and to publish the findings on a website (www.dableducational.org) 20. This enabled evidence-based information to be made freely available on the Internet to medical practitioners, consumers, such as patients, hospitals and doctors, and to the device manufacturing industry. Since its foundation, the Trust and its website have become the major international reference site for BP measuring devices, receiving over 1.6 million visits annually from over 5000 organizations in all parts of the world. The Trust does not rely on published reports of validation studies, but institutes its own strict review process to ensure that all validation studies have complied with the criteria of the validation protocols, and if there is doubt about the data provided or if there is evidence that the protocol requirements have been violated, the device is listed as ‘questionable’ rather then being recommended for clinical use. The Trust website also provides a reference library with over 400 publications on BP measurement and a peer-reviewed equivalence procedure to facilitate the posting of modified devices, if the manufacturer can meet strict criteria demonstrating that changes to a given device do not affect its measurement accuracy 21. Now the dabl Educational Trust has developed a program using the ESH International Protocol 2010 that will provide continuous on-line monitoring of a validation study from start to finish with requirements being checked as the data are uploaded during the study, and in the event of the protocol requirements not being fulfilled, an alerting message is sent to the investigator. This process will minimize protocol violations and also facilitate the posting of results on the website on completion of the validation study. Where to from here: validation laboratory accreditation? The validation process has undoubtedly improved BP device accuracy. A recent review showed that the ESH International Protocol has succeeded in expanding the validation procedure worldwide by three to four-fold compared with the period before its publication 14 Fig. 1. Successful though the process has been there are weaknesses and there is always room for improvement, especially in preventing validation procedures being influenced by fiscal considerations in a lucrative commercial market. Indeed the issue of trust, which is fundamental to all scientific endeavor, is as important in trials of drug efficacy as in the validation of BP measuring devices. In the former, improved design, randomization, blinding, power calculation, and so on, have solved several major methodological issues, and in the latter, experience from some 200 validation studies has likewise led to improvements in the methodology of the validation procedure, which have been incorporated in widely accepted protocols. However, trust depends on high-quality research and honest reporting of findings. Indeed, the infrastructure required for conducting a validation study might appear very basic – a quiet room, two mercury sphygmomanometers and three observers – but to this must be added the need for a clinical scientist with experience in BP monitoring research, superlative attention to detail and strict compliance with the validation procedure; these necessary prerequisites cannot be taken for granted and a process of quality assurance and mechanisms to prevent any conflict of interest are needed.Fig. 1Such indeed is the message in ‘Can we trust automatic sphygmomanometer validations?’ 2. Turner has highlighted a major weakness of the current validation process, namely reliance on peer-reviewed publications alone without formal certification of the centers performing validation studies, as is common practice with other measurement systems. Indeed the drawbacks of peer-review have been acknowledged by the dabl Educational Trust, which insists on having its own stringent checking system to detect protocol violations that are overlooked in published peer-reviewed papers. The provision of on-line validation will remove this concern, but it will not influence the integrity or competence of laboratories performing validation studies. The BHS has established a system for accrediting laboratories for device validation by ensuring that certain minimum criteria apply. A further advantage of this process is that devices submitted for validation by manufacturers will be allocated to laboratories by the BHS thereby preventing any influence from the manufacturer on the validation process (personal communication). However, Turner goes a step further in calling for formal accreditation of laboratories as is common with other measurement systems. This process is overseen by independent agencies, such as the United Kingdom Accreditation Service, a not-for-profit organization funded by accredited laboratories which is answerable to the International Bureau of Weights and Measures in Paris, France, the peak laboratory in the international measurement framework of cooperating national and commercial laboratories that provide reference standards and traceable calibration for all scientific, industrial and trade measurements worldwide. This process of accreditation is complex and presumably costly. The technicians, nurses and doctors involved in validation must receive specialized training; the test measurement system must be traceably calibrated according to an international standard 22; the laboratory must be shown to operate a quality system conforming to international standards 23; and accredited laboratories are required to participate in interlaboratory comparisons. The process detailed by Turner raises important questions that need to be considered carefully. Will the proposed accreditation process lead to more BP measuring devices being validated or could it have the effect of making the accreditation of validation laboratories so unworkable and expensive as to be counter-productive? How can the process be implemented taking into consideration the fact that BP device validation studies have previously been considered akin to research projects that have been conducted on a voluntary basis? How much will the proposed process add to the cost of device validation and who will pay? Whatever the answers to these questions, Turner has provided a stimulus for debate that should involve the international hypertension bodies, researchers interested in BP measurement, the regulatory agencies cited by Turner and the manufacturers of BP devices.