Caution warranted when torturing data until they confess

In the United States today, hospitals and health systems have reached a critical point with respect to their financial performance. While health care costs (particularly labor costs) have continued to rise rapidly, revenue streams have flattened and declined. As value-based reimbursement (being paid for quality and outcomes, not quantity of care delivered) continues to emerge as the dominant payment strategy, the tension between spending and income will increase. This tension is felt acutely in Departments of Pharmacy as, in the fee-for-service environment, pharmacy services are (by and large) not billable, and pharmacists are among the highest-paid nonphysician staff in health systems. While revenue generated through programs such as specialty pharmacy will protect some health system pharmacy departments, it has never been more important to show the value pharmacists provide through their direct patient care activities. This issue of JACCP includes a study by Hammond and colleagues measuring the value of clinical pharmacy interventions made in an intensive care unit of an urban academic medical center. They report cost avoidance in excess of $3 million over a 12-month period associated with interventions made by a pharmacist (half-time effort) and show a return on investment of over $24 for every $1 invested in labor costs for that pharmacist. Health system pharmacy leaders will no doubt receive these findings with enthusiasm and will be tempted to cite this study when justifying existing and new clinical programs in critical care. However, shortcomings in the costing methods used by these investigators must be recognized, and the results should be interpreted and used with caution. In this study, investigators documented and classified interventions as carefully as we have seen in any previous study. The duration of observation was relatively long (12 months, allowing for an evaluation of seasonality), the interventions were made by one practitioner (limiting confounding that often exists when interventions made by teams of pharmacists are documented), and interventions were adjudicated in a robust manner (including input from nonpharmacists). It is reasonable to accept the results as reported, at least in terms of the frequency and nature of the interventions made, although it should be recognized that the investigators only included interventions that were accepted by the physicians caring for the patients. However, the purpose of this analysis was to place a financial value on the potential impact of each intervention—documenting “cost avoided,” or what the cost of care would have been, were it not for the interventions made. The investigators have drawn from a recently published literature review, providing a qualitative evaluation of the financial impact of clinical pharmacy services on the intensive care unit (ICU) and emergency medicine areas. In that literature review, the authors (two of whom are also authors of this current study) pointed out the importance of quantifying the value of pharmacist interventions in the future. The estimates of financial impact from all but five of their 38 categories of pharmacist interventions were drawn from “studies” with very low methodological rigor—descriptive analyses and estimates based on expert opinion. The application of those cost impact estimates to any future study, including the current one, is simply inappropriate. It is possible to apply the cost impact estimates from the categories where the literature review had higher-quality data, and when that is carried out using the data from the current study, the net cost avoidance that can be more confidently claimed is approximately $800 000 (substantially lower than the $3.2 million reported but still a substantial benefit). This estimate includes four instances of medication route intervention when managing hypertensive crises (level IIA evidence, $79 000 total avoidance), 966 instances of antimicrobial therapy streamlining (level IB evidence, $582 034 total avoidance), and 63 instances of thromboembolism prevention initiation (level IIA evidence, $101 986 total avoidance). Another 37 interventions were performed in changing medication routes from intravenous to oral (level IIA evidence). The cost impact of those interventions is related to the actual mediation cost and is not reported, but it is reasonable to expect that the net savings in those cases would be relatively low. Again, even after limiting the analysis to interventions where we can be more confident in the size of the cost avoidance, the return on investment is still very strong—at least $5 to 6 per $1 in labor cost invested. However, the estimate claimed by the authors is a substantial overreach. All hospital and health system leaders recognize that they must be confident in the accuracy of their cost and benefit figures when developing business plans for new and existing programs. When relying on estimates and assumptions, there is still an expectation that their analyses are conservative—overestimating cost and underestimating potential benefits. Proposals undergo intense scrutiny by senior leaders who quickly recognize uncertainty and flaws in source data. Those using the study by Hammond et al should be cautious about relying too heavily on the estimated financial impact of interventions—they have shown clinical pharmacy services in critical care to be beneficial, but their reported $24-to-$1 return is simply not credible.