CORR Insights®: Substantial Inconsistency and Variability Exists Among Minimum Clinically Important Differences for Shoulder Arthroplasty Outcomes: A Systematic Review

Where Are We Now? As orthopaedic surgeons, our ability to assess and report on the successes of the patient care we provide and the surgeries we perform has undergone an important transition in the past 10 years. The documentation of “outcomes” has been an important component of these efforts. As we evaluate patient outcomes, many studies perform statistical assessments to try to minimize the likelihood that a chance event is misinterpreted as an actual treatment effect; many studies set a threshold p value of 0.05, although this is arbitrary, and there are situations when this threshold (and perhaps any threshold) should not be used [7]. We’ve come to realize that many statistical differences are too small (and sometimes far too small) for patients to perceive or care about. Large sample sizes—such as those that are sometimes employed in large-database studies or multicenter studies—can result in the detection of between-group differences that are too small to justify the kinds of interventions that surgeons use, all of which carry meaningful risk. Stated another way, we’ve come to focus more on whether observed differences, even if statistically validated, are large enough to care about; that is, are they clinically important? For example, in a large enough group of patients, an improvement in forward elevation of the shoulder after a specific operation of 5° might be detectable as a “real” change as opposed to a statistical aberration, but I doubt that most patients would select a specific shoulder procedure to obtain such an incremental benefit. The same general question might be raised vis-à-vis expensive interventions that deliver small or marginal returns [6]. The role of effect-size thresholds like minimal clinically important differences (MCID) [9], substantial clinical benefits [10], and the patient-acceptable symptom state [2, 5] have emerged as helpful methods of assessing effect sizes. Another important trend in terms of focusing on what matters to patients has been the understandable and justifiable move towards patient-reported outcome measures (PROMs); that is, the recognition that the patient’s assessment of improvement and satisfaction is far more meaningful than the surgeon’s assessment [1, 3, 8]. The combination of focusing on outcomes as perceived by patients and effect sizes rather than mere “statistical significance” have forever changed how we assess the success of the procedures we perform. Indeed, p values no longer rule the day. The outcomes of arthroplasty, especially shoulder arthroplasty, have focused extensively on these new measures of success. This has led to a proliferation of studies documenting varying values of disease and procedure-specific MCID thresholds for PROMs in various patient populations. In this month’s Clinical Orthopaedics and Related Research®, Kolin et al. [4] systematically reviewed 43 studies on the use of the MCID to assess the outcomes of total shoulder arthroplasty. They aimed to closely evaluate how MCIDs were used and reported, and to determine whether there was any meaningful consistency. Their analysis essentially opened a Pandora’s box of issues that should be addressed if MCIDs are to be a meaningful, reliable, and reproducible method of assessing outcomes. Their review identified three specific problems: There is substantial inconsistency and variability in the quantification and reporting of MCID values in total shoulder arthroplasty studies, the use of published MCID values ignores the value and necessity of calculating population-specific MCID thresholds, and the development of new MCID values (in 44% of the articles) did not follow a consistent protocol, but rather used differing methodologies, some of which may be more reliable than others. It is important to determine population-specific thresholds. The increasing emphasis on health equity requires us to recognize the differences in the populations we treat. Applying a pre-existing MCID value based on a specific population group may completely misrepresent outcomes in a different patient population. MCID values cannot be conflated between studies of differing designs because this will often introduce bias and lead to inaccurate conclusions. This is not a hypothetical concern. For those of us operating on patients, providing care, and discussing the potential outcomes of the treatment we provide, we need to know that the interventions we offer are likely to deliver benefits large enough to please the patients we care for. Where Do We Need To Go? As the authors [4] point out, the use of MCIDs is based on identifying the smallest difference in an outcome measure that a patient will perceive as a meaningful change. In 56% of the articles reviewed, MCIDs in new patient populations were compared with previously calculated MCID thresholds that may have been derived form a completely different patient population. This highlights the need for population-specific MCID values that may vary based on age, gender, race, geographic location, and other factors. Our first step in addressing the problem is to develop a standardized approach to reporting MCID values. If authors wish to use established values, there should be justification based on the population being studied. If a suitable established MCID value is not available, then a new MCID value should be determined. Furthermore, a standardized approach to analyzing MCID values and constructing population-specific MCID thresholds should be developed. One of the issues the authors [4] identified was that five different categories of MCID calculations were used: anchor-based, distribution-based, alternative, arbitrary, and not specified. My preferred method for determining the MCID is either anchor-based or distribution-based. Although the anchor-based method is favorable and often used, the ideal anchor question has not been identified. Distribution-based approaches are convenient (and may be the only possible approach in some large-database studies), but they’re not based on patients’ actual perceptions of the treatment benefit, which is a clear shortcoming. Going forward, guidelines for when and how to use each technique should be established. Although this study [4] only focused on the MCID as the key outcome measure, the use of other outcomes including substantial clinical benefits and the patient-acceptable symptom state should also follow the same approach. I think the next 10 years will bring a proliferation of shoulder arthroplasty outcome studies that will make these issues even more important, not less. How Do We Get There? Now that we know where we to need to go, getting there will be a formidable task but not an impossible one. This will require the standardization of outcome reporting that should be initiated by the editorial leadership of orthopaedic journals. Just as we have a standardized conflict of interest form developed by the International Committee of Medical Journal Editors and a standardized method of reporting Levels of Evidence, a similar approach can be used to address the issue of outcome reporting. A consortium of the major orthopaedic journals including subspecialty journals should convene to establish guidelines for reporting outcomes. Admittedly, this is a complex issue, but the development of such guidelines will have an important impact. If standardization is accomplished, followed by widespread acceptance, it would then be possible to better understand the outcomes reported in the studies we read and to truly be able to compare one study to another. The benefit to us as orthopaedic surgeons would only be surpassed by the benefit to the patients we treat.