Quality improvement studies – pitfalls of the before and after study design

From a clinical perspective, the problem of postoperative residual paralysis is rather intriguing from several aspects. First and foremost is that it is a difficult phenomenon to define and quantify 1. Far from being a straightforward response, recovery from pharmacological neuromuscular blockade occurs at different rates for different skeletal muscle, the order of which can vary from one individual to another. Clinical assessments of muscle strength are subjective and tend to be unreliable for assessment of residual paralysis 2. Objective assessment parameters are indirect and non-linear, being an indicator of receptor occupancy and not of muscle strength. For example, the train-of-four ratio (TOF) provides discriminating information of the final 30% of receptor occupancy but that does not reliably predict what may be happening in the pharyngeal muscles 3. We tend to monitor neuromuscular function at sites that are accessible but have minimal clinical consequences, and muscles in which weakness can lead to significant morbidity are not readily accessible. The pharmacology of neuromuscular blockade reversal is also not straightforward 4. Before the availability of sugammadex, there were no direct agents for antagonising non-depolarising neuromusular blocking agents (NMBA). Rather, we relied on the antagonism of cholinesterase to increase acetylcholine at the motor endplate in order to affect muscle cell depolorisation. Being reliant on an enzyme system that is not specific for the nicotinic receptor, the cholinesterase inhibitor is prone to a ceiling effect 5 and can induce muscinarinic adverse effects. In addition, neostigmine is known to have a paradoxical effect by increasing residual paralysis when given in excess to what is required. Sugammadex may have gone some way to circumvent these pharmacological complexities. In addition, this agent enables the use of large doses of rocuronium at induction to produce rapid onset and profound paralysis, even for short procedures, without the inconvenience of delayed or inadequate reversal of neuromuscular blockade. This can result in superior operative conditions for different types of surgical procedures 6, 7. On the other hand, the agent is expensive, not widely available and is only efficacious for aminosteroid neuromuscular blocking agents. Therefore, the combination of neostigmine plus an antimuscarinic agent remains the mainstay pharmacotherapy for the antagonism of NMBAs in the foreseeable future. Standardising the management of reversal of muscle paralysis is no mean feat, as is any other endeavour that involves convincing a group of anaesthetists to unify their practice! In addition to it being a multistep process of assessing the TOF ratio and then undertaking the calculation of the correct dose, there may be some extra, hidden difficulties with this challenge. If seeing is believing, it may well be the case that anaesthetists are not ‘seeing’ residual muscle paralysis, as a previous survey highlights a gross discrepancy between what is estimated by the attending anaesthetist and what is objectively being measured 8. Perhaps it may be the case that those muscles that when weak would contribute to delayed pulmonary complications are ‘hidden’ from sight (e.g. pharyngeal muscles) and the manifestations of weakness are subtle and difficult to observe. One would have to perform an additional measurement that involves the direct application of a neuromuscular function monitor in order to detect weakness in these muscles. Most pulmonary complications are not obvious in the immediate peri-operative period and thus there is not a strong cognitive link between residual neuromuscular blockade and associated morbidities. Some clinicians may even be inherently sceptical as to whether TOF monitoring can actually predict and/or prevent postoperative pulmonary complications. It is against this background that Rudolph et al. 9 conducted their study to improve the peri-operative management of neuromuscular blockade and to evaluate its effects on postoperative pulmonary complications. Randomised controlled trials (RCT) are considered the most appropriate approach for assessing the effectiveness of interventions in the ideal world, but not all interventions can be practically assessed using this method. In addition, the external validity of clinical trials are often poor, and findings from RCTs can often not be generalised to real-world settings 10. Observational studies can overcome some of the deficiencies of a clinical trial, but confounding variables and difficulties in establishing causation limit the conclusions which can be drawn from such trial designs. This current study 9 is a prospective observational study comparing the incidence of major postoperative pulmonary complications before and after implementation of a quality improvement initiative. The primary outcome was a dichotomised composite measure of pneumonia, respiratory failure, pulmonary oedema and/or tracheal intubation. It would not be ethically acceptable to randomly select patients to be treated with, or without the quality improvement initiative, and so a quasi-experimental approach was used. Quasi-experiments are studies which aim to evaluate interventions without the use of randomisation and, like an RCT, aim to demonstrate causality between an intervention and an outcome 11. The lack of randomisation is the major weakness of such a study design, but associations identified in quasi-experiments meet the requirement that the outcome of interest occurred after the intervention, and therefore, a quasi-experiment is not simply an observational study 12. Internal validity of an experimental outcome is defined as the degree to which observed changes in outcomes can be attributed to the intervention, and when applying a quasi-experimental design, it is important that threats to internal validity are reduced 12. The first threat to internal validity is selection bias, where systematic differences in conditions in respondent characteristics could also cause the observed effect. The investigators in this current study have reduced this threat by conducting what in essence are two analyses, one on the whole cohort and one on a propensity matched cohort; this ensures that the patient characteristics in the pre- and postintervention cohorts are well matched. The ‘calliper’ used by the investigators for matching is a strict 0.1. This means that the maximum tolerated difference between matched subjects in a ‘non-perfect’ matching intention is set at 0.1 standard deviations of the propensity score. Results from the propensity-matched group confirmed the results seen in the whole cohort, which strengthens the finding that the neostigmine dose significantly decreased after the intervention. Another threat to internal validity, known as maturation, is when naturally occurring changes over time are confused with an intervention effect 13. The investigators have handled maturation by analysing the data using an interrupted time-series design. A time series is a continuous sequence of observations on a population taken repeatedly over time. In an interrupted time-series study, a time series of the outcome is used to establish an underlying trend. The current study took measurements of weekly mean neostigmine dose per case from 15 months before intervention to establish trends in changes. The hypothetical scenario, under which the intervention had not taken place and the baseline trend continues unchanged, is known as ‘counterfactual’, and the counterfactual scenario provides the comparison for the evaluation of the impact of the intervention. Figure 1 is a reproduction of Fig. 4a in the current study 9, with the added counterfactual line, showing the expected trend of mean neostigmine use had the intervention not taken place. As can be seen, the counterfactual line does not intersect with the post-intervention line, showing that post-intervention use of neostigmine did indeed decrease 12. When applying the principles of interrupted time series, it is important to remember that it would only be relevant if the intervention was not introduced gradually. The current study 9 did not strictly adhere to this requirement, because the change in neostigmine vial size was instituted 3 months before the department-wide educational activities and distribution of cognitive aids. As can be seen in Fig. 1, there is a decrease in neostigmine use after week 66, when smaller aliquot sizes were introduced, and a further decrease after week 79, when cognitive aids and other educational activities were instigated. Instrumentation, where the nature of a measurement may change over time and conditions, is another threat to internal validity. ‘Instrumentation’ may be an issue in the current study, where the ‘instruments’ are the clinicians documenting the neostigmine used and TOF ratio. During the intervention, clinicians were offered a financial bonus for documenting TOF ratio before neostigmine use, and this financial incentive may have resulted in superior documentation. A summary of the steps recommended for the implementation of a quality improvement study is given in Table 1. Analysis of the results should occur contemporaneously with the conduct of the project Interrupted time series, which takes into consideration previous performance trends, should be used A scatter plot of the time series, which can help to identify underlying trends, seasonal patterns and outliers, should be constructed. Descriptive statistics should also be given Testing and identifying improvements and addressing methodological issues: In conclusion, this study has shown that a quality improvement intervention to improve the peri-operative management of neuromuscular blockade does reduce the number of postoperative pulmonary complications, but a financial incentive may be required to encourage the uptake of this new practice. SC is statistical advisor to Anaesthesia. No other competing interests declared.