Pitfalls of clinical practice guidelines in the era of broken science

Clinical practice guidelines have become an instrumental tool to guide clinical practice. Their popularity is mainly driven by physicians’ desire to have easy access to recommendations that may facilitate their daily work. But equally, the public and legislators have a certain interest in guidelines to achieve the best possible care because of mounting medical costs, availability of a plethora of newly developed expensive devices and technologies, a transformation of healthcare delivery, a changing demographics with rising number of elderly in need of treatment and finally a substantial variation in the quality of care. Our daily reality is one of clinicians being constantly overwhelmed by the quantity of reported evidence due to a mounting number of publications which makes guidelines more popular due to our need for aggregated and summarised evidence to guide our clinical decision-making. Theoretically, adherence to clinical guidelines should reduce variation in care and unnecessary cost while increasing quality and improve cost value measures.1 The American Institute of Medicines (IOM) has defined five major purposes for clinical practice guidelines: first, assisting patients and practitioners in clinical decision-making; second, education of individuals or groups; third, assessing and ensuring the quality of care; fourth, allocation of healthcare resources; and fifth, reducing the risk of legal liability of negligent care.1 The number of guidelines globally is increasing dramatically. For instance, the Guidelines International Network (www.g-i-n.net/about-g-i-n) which was founded as a global network in 2002 currently comprises 105 organisations representing more than 53 countries from all continents. This network, which supports evidence-based healthcare and improved health outcomes, now comprises more than 6400 guidelines in its International Guideline Library. The US National Guideline Clearinghouse (www.guideline.gov) and the US Trip Medical Database (www.tripdatabase.com) cover between 1500 and 3000 healthcare-related guidelines, whereas the National Institute for Health and Care Excellence (www.nice.org.uk) currently holds more than 1700 guidelines. However, beyond these major players and large scientific organisations, it often appears as if different guideline groups are able to review the same clinical condition and reach conflicting conclusions.2 This mass-production will naturally erode the validity of these guidelines and reduce trust by healthcare providers and administrators.3,4 But how did we get here? Historically, much of the need for guidelines is derived from the recognition of widespread practice variation and exploding healthcare costs dating back to 1970s and 1980s in the United States. This combined with real concerns among physicians and institutions for litigation in case of adverse outcomes led to promotion of guidelines.5 Although clinical guidelines may appear beneficial for our patients as a group, they may not always be appropriate for the individual patient and certainly not for patients with various comorbidities. The increasing prevalence of patients with multiple chronic conditions with rising complexity raises concerns about the appropriateness and applicability of clinical practice guidelines for patient management as most guidelines currently are designed with a single chronic condition in mind and are not adapted to comorbidity-related issues.6 In addition, the routine approach of guidelines is often in conflict with the concept of personalised medicine and shared decision-making.1 Methodologically rigorous guidelines are massively resource-demanding and time-consuming which may ultimately delay dissemination of urgent guidance even if they are considered methodologically solid. What is the cause for concern is the degree of trustworthiness of guidelines. Do they improve outcomes and maximise resource use? How do we decide if a clinical practice guideline is trustworthy and how do we raise standards? How do we act when clinical guidelines provide conflicting recommendations based on the same evidence? IOM has proposed a set of standards for trustworthiness of guidelines that are in line with the policies of the European Society of Anaesthesiology (ESA).7–10 In general they consist of eight elements: A transparent process of guideline development and funding; Declaration of conflicts of interest where ideally none or at most only a small minority should have conflicts and the chair and cochair should not have any; The composition of guideline development groups should include active participation of methodology experts (as of now mandatory for ESA guidelines), clinicians and ideally also representatives of stakeholders and affected populations; Systematic synthesis of evidence through systematic reviews of the literature in accordance with existing standards such as Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)11 and recommendations by Cochrane Collaboration;12 Grading of evidence and providing strength of recommendation with explanations of the rationale behind each recommendation, summary of evidence for benefits and harms using GRADE Methodology (www.gradeworkinggroup.org); Providing easy to use recommendations, flow-charts and other means that can facilitate adherence and disseminate the message; External review by various stakeholders and the public; and Update of guidelines with new emerging evidence. It is important to emphasise that despite an apparent legal importance of guidelines in many settings for litigation purposes, guidelines are not meant to devise reimbursement policies, promote performance measures, act as legal precedents or to be used as measures of certification or licensing or public reporting.13 They are meant to promote care! However, despite the best of intentions to raise standards of guidelines, the recommendations of even high-quality guidelines may be flawed. As described above, systematic reviews are a cornerstone of guidelines as they provide a summary of evidence. However, there is an alarming methodological shortage in many of the published systematic reviews that increases the risk of false conclusions and recommendations.14,15 Thus, poorly conducted systematic reviews may cause a great systemic flaw and bias. Academic and scientific organisations and public health providers may formulate guidelines that may misinform and misguide clinicians, exhaust taxpayer's money and provide little improvement of health and may even put patients at danger. Yet, there seems to be an apparent ignorance or perhaps indifference among reviewers and editors of scientific journals to methodological issues and shortcomings of systematic reviews in which conclusions are taken at face value.16 As an example, despite ongoing efforts to raise standards even among high-impact medical journals, Cochrane systematic reviews are consistently rated to have higher methodological quality, but remain cited less frequently than non-Cochrane systematic reviews of lower methodological quality.17 With an annual 3% increase in number of systematic reviews published in PubMed exceeding the number of published randomised clinical trials,17 we are witnessing nothing less than a tremendous rise in production of misleading, often unnecessary, flawed and high risk of bias systematic reviews and meta-analyses. This is despite efforts of many grass-root organisations such as the Cochrane collaboration to increase the methodological quality of systematic reviews by introduction of tools such as PRISMA guidelines,18 PRISMA-protocol guide19 and the Methodological Expectations of Cochrane Intervention Review.20 But despite all these efforts the majority of current research effort seems wasted because of failed priorities and methodological and design shortcoming of trials and systematic reviews. And let's face it. Even despite best intentions and highest methodological standards and rigour, systematic reviews and guidelines are no better than the trials and publications they include. How do we, for instance, address the fact that industry sponsorship significantly affects research outcome? In a recent Cochrane systematic review, industry-sponsored studies were found to have 27% more favourable efficacy results relative risk 1.27; 95% confidence interval (CI) 1.17 to 1.37.21 In another recent publication, financial ties of principal investigators of clinical trials were independently and strongly associated with positive clinical trial results (odds ratio 3.23, 95% CI 1.7 to 6.1). Thus, the findings of systematic reviews including these biased industry-sponsored trials, will naturally also affect clinical practice guideline recommendations if their methodology and conclusions are not scrutinised by guideline groups.22 Another major challenge in regard to publication of these poor quality systematic reviews is the issue of conflict of interests and lack of methodological knowledge of peer reviewers. We are often faced with little spare time, and the journals often demand quick assessment of articles with unrealistic time limits and provide little incentives for the peer-reviewers. Furthermore, there are no formal requirements for the peer-reviewers to have any statistical or methodological knowledge about systematic reviews, meta-analyses, GRADE or guideline-methodological requirements. Adding to the above is the issue of scientific fraud and detection of fraudulent or manipulated trials that are included in systematic reviews and clinical practice guidelines. The Committee on Publication Ethics has since 1997 formulated standards and guidance for editors for the detection and retraction of fraudulent research but the problems with scientific fraud remains far from resolved and certainly when faced with an explosion in the number of predatory fake online journals, one should be even more alert.23 As the academic and financial incentives of scientific publications for researchers and research institutions remain high, eradication of scientific fraud remains unrealistic unless we embark on a path of international collaboration trying to raise standards and moving beyond the notion of ‘publish or perish’. We need to rethink the entire publication and academic system. We have to move beyond our obsession with statistical significance and P values. We should acknowledge and address the issue of reproducibility of trial findings and data sharing. By some estimates, less than 50% of findings from basic science research from academic institutions can be reproduced in an industrial setting.24 These very same publications lay the foundation for design of new drugs and clinical trials that often fail or show little effect or may even harm the very same patients we want to help. Not to mention the misuse of resources. Moving up the latter, the quality of a large portion of the published clinical trials remains poor. We are witness to an increasing number of underpowered small clinical trials. By just merely searching on PubMed and using the keyword ‘pilot studies’, it becomes evident that during the last 30 years, there has been an astonishing rise in the number of pilot studies. As the medical literature is being drowned by these biased, poor quality, inadequately designed and underpowered trials, often ‘disguised’ as pilot studies, the systematic reviews and clinical guidelines which are based on the very same trials will naturally be biased as well. As recently reported in a BMJ publication, the median number of trials in Cochrane reviews is between six and 16 with a median number of patients per trial of about 80. Thus, most meta-analyses despite methodological rigour will suffer from sparse data and remain underpowered with a high risk of random error resulting in false, often positive, conclusions.25 The impact of inadequate methodological standards in research has not only direct impact for patient care but poses a great financial burden for our societies. Some recent estimates indicate that as much as 85% of global health research is wasted because of inadequate methodology and poor study design and conduct.26 Not only is this figure breath-taking, but given the fact that 200 billion US dollars annually is spent on health and medical research, this waste of estimated 170 billion US dollars globally is shocking. But how should we address the methodological deficit in science? One of the main expert opinion makers, Professor John Ioannidis, has recently proposed a list of research practices which could help increase the proportion of true research findings. They include requirement for large-scale collaborative research; adoption of replication culture and reproducibility practices; registration of all steps of trial conduct (protocol, analyses, data set and results); sharing of data, containment of conflict of interest for sponsors and authors; higher statistical and methodological standards; improvement of study design standards, peer-review, reporting and dissemination of research; and finally better training of scientists in methods and statistics such as editors, peer-reviewers, trial investigators and authors.27 Clinical guidelines often have immense impact when issued by major scientific organisations as they act as a standard of care which may be used to devise national and local protocols, measure physician performance, evaluate adherence to standards, to carry out legal and insurance coverage decisions and decide on the choice of drugs to be purchased and used.28 Moreover, they can also be used as ‘expert testimony’ in cases of litigation and malpractice. Furthermore, we should in no way overlook the impact of conflicts of interest as despite various attempts to limit or reduce the conflicts of interests among guideline authors and sponsors, most guideline authors have conflicts of interests that may reduce the reliability of guideline recommendations.7,29,30 Neither should we turn our blind eyes towards the impact of industry influence on clinical guidelines.31 As a scientific society, much of our credibility depends on the very same guidelines that we draft and promote and the founding research that we carry out at our institutions. We have an urgent obligation to raise the methodological standards of science in general. Our goal should be to increase the methodological standards of our guidelines in accordance to IOM set of standards for trustworthiness. Because the sad reality is that only a fraction of the published clinical guidelines live up to these standards.31–33 Moreover, for the majority of guideline authors, the magnitude of their industry tie would disqualify them from authorship if measured against IOM standard.34–36 We may never be truly able to eliminate the issue of conflicts of interest or eliminate scientific fraud, but as a scientific society, the ESA aims to improve standards and we recognise that it is time to address the issue of broken science.37 Acknowledgements relating to this article Assistance with the Editorial: none. Financial support and sponsorship: none. Conflicts of interest: none. Comment from the Editor: this Editorial was checked and accepted by the Editors, but was not sent for external peer-review.