The Endocrine Society Clinical Practice Guidelines: A Self-Assessment

In 2005, after many years of discussion about whether or not The Endocrine Society (TES) should produce Clinical Practice Guidelines (CPGs), TES Council approved a carefully designed program to produce 2 to 4 high-quality CPGs each year. TES has published 20 CPGs to date (1), and several more are expected to be published in the next 2 years. Since its inception, the CPG program has been unique in many ways. First, a highly regulated pathway on the route to publication (Figure 1) permits the selection, review, and vetting of CPGs at multiple levels, including a final peer and editorial review at the Journal of Clinical Endocrinology and Metabolism (JCEM). Second, the guidelines have been selected and produced without support from pharmaceutical or device-manufacturing companies. Third, all CPG expert panels have included at least 1 international member. Fourth, all CPGs have been translated into patient versions. Fifth, the CPGs are routinely reviewed and updated every 3 years. Finally, TES adopted the Grades of Recommendation Assessment, Development and Evaluation (GRADE) approach for evaluating the evidence and producing recommendations (2). The Endocrine Society's process of Clinical Practice Guideline Development. The GRADE Working Group began as an international effort to resolve confusion about how to best and most clearly rate evidence and express recommendations. Many governmental organizations and professional societies use the GRADE approach, including the Agency for Healthcare Research and Quality, the American College of Physicians, and UpToDate in the United States; the National Institute for Clinical Excellence and the British Medical Journal in the United Kingdom; and the World Health Organization (3). Although there are multiple systems of grading evidence, the GRADE method is the only one that rates the strength of the recommendation (Strong or Weak) based on a balance of benefits vs harms, the confidence in the magnitude of estimated effect on an outcome, and the weighing of patient values and preferences. Although this method allows for Strong recommendations to be made with low (L) or very low (VL) levels of evidence and vice versa, this should not be a frequent occurrence. TES engaged the Knowledge and Evaluation Research (KER) Unit at the Mayo Clinic to provide methodological assistance to each of the CPG expert panels, whose expertise generally is in clinical medicine and not in methodology or epidemiology. In this issue of JCEM, members of the KER Unit have undertaken a unique exercise—assessing the appropriateness of 357 recommendations contained in all 17 CPGs that were published by TES between 2006 and 2011 (4). Their major finding was that 206 of the 357 recommendations were Strong, of which 121 were based on L/VL levels of evidence. Of the 121, 53 were in categories such as “sensible alternatives do not exist” or recommend “additional research.” KER Unit members then retrospectively applied a recently published schema by the GRADE Working Group of paradigmatic situations where Strong recommendations with L/VL evidence might be appropriate (5). Using this new taxonomy of paradigmatic situations to determine the appropriateness of the remaining 68 Strong recommendations that were based on L/VL levels of evidence, they found that it was appropriately applied to 35 recommendations and inappropriately applied to 33 (6–16). Supplemental Table 1 (published on The Endocrine Society's Journals Online web site at http://jcem.endojournals.org) shows those deemed to be inappropriate (personal communication from Dr Juan Pablo Brito). A number of important questions arise as a result of this self-assessment. First, how well is the CPG program of TES doing? It would appear that the CPG program is doing extremely well overall. Brito et al (4) assert that 27% (33 of the 121 Strong recommendations with L/VL evidence) of the recommendations were unjustified. However, it may be more informative to look at these 33 recommendations more globally using either 357 (the total number of recommendations) or 306 (357 − 53 “sensible alternatives” or required “additional research” recommendations) as the base. Thus, the percentage of “unjustified” recommendations is 9 or 11%, respectively. Another approach is to assess the number of “unjustified” recommendations in each guideline. For example, in 2008, there were 5 guidelines published with a total of 106 recommendations, of which 12% were “unjustified” according to the new GRADE schema (5 of 22 in “The Diagnosis of Cushing's Syndrome”; 3 of 27 from “Primary Prevention of Cardiovascular Disease and Type 2 Diabetes in Patients at Metabolic Risk”; 4 of 12 from “Case Detection, Diagnosis, and Treatment of Patients with Primary Aldosteronism”; 0 of 17 from “Evaluation and Treatment of Hirsutism in Premenopausal Women”; and 1 of 38 from “Prevention and Treatment of Pediatric Obesity”). The diversity in the frequency of “unjustified” recommendations may be due to the fact that certain expert panels had more assistance from the methodologist than others, that for some CPGs it is inherently more difficult to develop clear recommendations, or that the evaluation by the GRADE Working Group's new schema was unequally applied. It is likely that there are elements of all 3 explanations underlying the findings of Brito et al (4). Second, does the analysis by Brito et al (4) diminish the validity or the clinical importance of those 33 Strong recommendations based on L/VL levels of evidence? At the very worst, these recommendations represent “expert opinion” that, in the absence of high-quality evidence, can provide useful and in some instances the only feasible form of guidance for making difficult clinical decisions by cutting through the cacophony of conflicting, weak, or nonexistent data. Many of the 33 recommendations based on L/VL levels of evidence either addressed “safety concerns” or were related to the interpretation or application of laboratory tests. In addition, it should be noted that few laboratory tests have undergone extensive testing in large randomized trials. Historically, there has been a paucity of population-based, rigorously derived reference ranges for most hormones and analytes. Although TES has recently initiated efforts to generate and harmonize population-based reference ranges for several hormones, it remains the case that the thresholds for defining hormone deficiency or hormone excess for most endocrine disorders are necessarily based on either a L/VL level of evidence or expert opinion. Third, do systematic reviews help expert panels make the correct recommendations? Brito et al (4) express concern that strong recommendations are made with “evidence yielding limited confidence in the estimates of effect about the available options.” It might be expected that systematic reviews would ameliorate this legitimate concern. However, Brito et al (4) found that systematic reviews were used in 14 of 125 (12%) Strong recommendations that were made with L/VL quality of evidence. Weak recommendations were made with L/VL quality of evidence to a similar degree (21 of 122, or 19%). Thus, it is unclear whether using more systematic reviews, which are often strongly considered in evaluating the evidence, would be helpful in avoiding inappropriate recommendations. Fourth, how does TES compare with other organizations? It would appear that TES compares quite favorably with other organizations in the quality and validity of CPGs. For example, it is reassuring that the rate at which CPGs of TES apply Strong recommendations with L/VL evidence is similar to the American Association of Blood Banks and the Society for Vascular Surgery, which made Strong recommendations with L/VL evidence 64 and 65% of the time, respectively, using the GRADE system, although their “appropriateness” was not addressed (17, 18). Indeed, this may be viewed as an inherent flaw of the GRADE approach or in its appropriate application. Comparing guideline development of TES to that of other endocrinology organizations is difficult because of the differences in methodologies used. No other endocrinology professional society uses the GRADE approach, and the criteria for development and approval of guidelines of other societies are less rigorous and/or less transparent (19, 20). Fifth, how can TES improve the quality of its guidelines? Quality improvement is a goal of any organization. Indeed, the KER Unit's effort is an important and useful attempt to assess and improve on their own performance as well as to determine how the GRADE system can best be used to make clinically valid recommendations. TES whole-heartedly embraces the evolving nature of the GRADE guideline development process and is committed to closely integrating methodological advice into every guideline. By recognizing paradigmatic situations that better define when Strong recommendations with L/VL evidence are appropriate, TES should be able to continue to be in the vanguard of producing the highest quality guidelines for our members and their patients. Evidence-based practice involves complex decision-making that is based not only on the available evidence but also on patient characteristics and preferences, values of the patient and physician, and ever-changing uncertainties. It is certain that high-quality evidence will not be available in all clinical situations, and in situations where high-quality evidence is not available, expert opinion and careful synthesis of low-quality evidence will continue to appropriately guide clinical practice. Disclosure Summary: R.A.V., S.B., and K.A.M. are the first, second, and current chairs of the Clinical Practice Guideline Subcommittee, respectively. The opinions expressed in this paper reflect the personal views of the authors and not the official views of the United States Army or the Department of Defense. Clinical Practice Guideline Grades of Recommendation Assessment, Development and Evaluation Knowledge and Evaluation Research low very low.