Is Risk-Adjusted Mortality an Indicator of Quality of Care in General Surgery?

In Brief Objective(s): Profiling of hospitals using risk-adjusted mortality rates as a measure of quality is becoming increasingly frequent. We sought to determine the validity of this approach by comparing the risk-adjusted predicted mortality to the findings of concurrent peer review and retrospective chart review of deaths that occur on a general surgery service. Methods: Consecutive patients admitted to a busy general surgery service from January 2000 to January 2006 were prospectively entered into the Surgical Activity Tracking System. Rigorous, systematic peer review was performed concurrently by service members on all deaths. Adjudication was later validated by an independent senior surgeon. Three methodologies of risk adjustment (University Health Consortium, Physiological and Operative Severity Score for the enUmeration of Mortality, and the Charlson index) were used and compared the “excess mortality” predicted by each to the number of potentially preventable deaths determined by peer review. Results: A total of 9623 patients were admitted and 75 died (0.7%). University Health Consortium and Physiological and Operative Severity Score predicted an excess mortality of 62 and 65 deaths, respectively; Charlson predicted that 73% of the cohort would be dead in 1 year. Concurrent and retrospective peer review found that death was potentially preventable in only 22 and 21 patients, respectively. Conclusions: Peer adjudication and extensive clinical review adds much to the analysis of an adverse outcome, similar to the “black box” in an airplane crash. Although methods of risk adjustment may be helpful in identifying patients for peer review, they should be used for internal process improvement and not published as metrics of hospital or provider performance. Risk-adjusted mortality is commonly used as an index of hospital performance, implying that when the observed death rate exceeds that predicted by risk adjustment, there are provider-related or system-related problems with the process of care. Compared with rigorous concurrent and retrospective peer review, risk adjustment does not consider many factors vital to the assessment of whether or not a death could have been prevented.