CUSUM: a tool for ongoing assessment of performance

Historical examples of poor patient care going unnoticed over many years at a center or by a single doctor, such as the Bristol Royal Infirmary case1 or the Harold Shipman drama2, have pointed out the need for methods that measure clinical performance continuously. Health authorities have become resolute for such systems to be implemented in clinical settings in order to improve patient safety, develop the quality of the care provided, and reduce costs3-5. In this context, methods for statistical process control, developed in industry to monitor the quality of manufactured products, have proved very useful6. The value of these methods for analyzing medical data was first reported in medicine in the 1970s7, 8. Subsequently, their power to detect suboptimal clinical performance when it occurs so as to shorten the time to decision (continuation or modification of the monitored process) has been confirmed in various settings, with a particular benefit to interventional and surgical processes9-13. Recently, these methods have been extended to monitoring the performance of trainees during the learning of a process to indicate when competency has been reached14. Such a process is illustrated in this issue of the journal, in an article from Rozenberg et al.15. The most commonly used process control methods in medicine are the sequential probability ratio test, the exponentially weighted moving average and the cumulative sums (CUSUM) method16, 17. Of all available methods, the CUSUM test has received more attention in the medical literature owing to its simple formulation and a very intuitive representation6, 18. After each procedure the CUSUM tests whether the process under scrutiny is ‘in control’, that is if the process is performing at an acceptable level; if this hypothesis is rejected, the process is said to be ‘out of control’, and an audit should take place to determine the reasons for this inadequate performance and implement corrective actions. In the present statistical opinion we expose the principles behind statistical process control and discuss their possible value in assessing the quality of ultrasound imaging processes in obstetrics and gynecology. We present a worked example of the CUSUM test used as a quality control procedure to continuously monitor the performance of sonographers as they assess nuchal translucency (NT) in the first trimester of pregnancy. Control charts were originally designed to monitor the quality of manufactured products12. In medicine they have been used to monitor a wide range of outcomes such as the success of a surgical procedure13, an infection rate19 and the availability of beds in a hospital20. The use of statistical process methods requires being familiar with a few concepts that originated from quality control, as presented below. In any medical process, regardless of how successful and reproducible it has proven, a certain amount of variability around an established value, called the target, is expected. This natural variability, sometimes referred to as background noise, responds to many unknown and unavoidable small causes called chance or common causes. A process under the effect of these common causes of variation only is said to be ‘in control’. On the other hand, a process may be affected by other more important causes that are not part of the chance pattern, called assignable or special causes, and in this case the process exhibits more variability around, or a deviation from, the target and it is said to be ‘out of control’. Carefully conducted and highly reproducible medical processes will probably avoid the effect of special causes for a long period of time, during which the process will be regarded as in control with ordinary variations around the target. Nonetheless, the basis for implementing a monitoring procedure is that one should never consider that a process will stay in control forever, and expect that at some point a shift to an out-of-control state will occur owing to the sudden influence of a special cause. The basis for using CUSUM tests to analyze sequential observations is that they are not usually amenable to standard statistical analyses because of the inflation of the Type I error probability if successive tests were performed. For instance, if one computes a test after every new observation and compares the observed mean or proportion of these samples with the target value, the risk of reporting a false-positive difference between the two values will dramatically increase with the number of tests. The CUSUM test computes, at each time t, a score St defined by St = max(0, St−1 + Wt), with S0 = 0 and Wt the sample weight. Wt is a measure of the deviation of the observation from the target and depends on the endpoint monitored (Appendix). For continuous outcomes, the further the observation is away from the target the larger the Wt; for binary outcomes, a failure will usually result in a positive value for Wt and a success will yield a negative and smaller value for Wt. Therefore, the more the process deviates from the target the more St increases. At each time t the CUSUM tests the null hypothesis that the process is in control against the alternative hypothesis that the process is out of control; practically, if St is equal to or greater than a limit h, the CUSUM test rejects the null hypothesis and the process is deemed to be out of control. As long as the CUSUM test remains below the limit h, the null hypothesis cannot be rejected and the process is considered as in control. The CUSUM test has a very intuitive graphical representation where the scores St are plotted against time or, more frequently, the successive procedures t. Two limits, an upper (or lower, depending how the CUSUM test is designed) control limit at h and an absorbing barrier at zero that cannot be crossed, are plotted on the graph. If St reaches h or above, an alarm is raised and the CUSUM test signals that the process is out of control. If at t, the CUSUM score St is between zero and h, namely the continuation region, the null hypothesis is not rejected at this point and monitoring continues. Because the CUSUM score cannot cross the holding barrier at zero, the null hypothesis is never accepted and monitoring continues until either an alarm is raised or one runs out of observations. Monitoring of the learning curve using LC-CUSUM (Learning Curve CUSUM), as done by Rozenberg et al.15, requires a close but nonetheless different approach. In this particular case we want to determine whether a process has reached a predefined level of performance. It presumes that the process is not in control at the start of monitoring (the trainee is not proficient) and signals when the process can be considered to be in control, in other words that the trainee has reached the acceptable predefined level of performance. Therefore, the hypotheses are inverted for the LC-CUSUM, with the null hypothesis that the process is out of control and the alternative hypothesis that it is in control. Advances in methods for quality control of fetal measurement have largely benefited from the NT experience21-24. Quality assessment in prenatal ultrasound imaging has proven useful in the first trimester of pregnancy. During the 11–14-week scan, screening with NT measurement allows identification of a group of patients at high risk, who are offered an invasive diagnostic test25. Because the latter carries a risk of miscarriage26, it has appeared critical that inclusions in this high-risk group should be controlled and contained to around 5% of the entire population, allowing appropriate screening efficiency and an acceptable fetal loss rate22, 25. The comparison of NT measurements to reference values has become an established quality control process of ultrasound examination in the first trimester of pregnancy22, 25. Such a process is usually performed retrospectively by external reviewers to detect inaccurate sonographers. However, at the time of audit, a certain number of mothers may have been offered fetal karyotyping unnecessarily, whereas some fetuses with Down syndrome may not have been detected prenatally. We describe how the CUSUM could be implemented prospectively in order to minimize the time to making the decision that inadequate performance has occurred. Consider a center in which a sonographer routinely performs antenatal ultrasound measurements of NT. Each measurement is compared to a reference value, adjusted for gestational age, and the difference between the observed and expected value (delta NT) is used to determine the likelihood of Down syndrome of the fetus27. We know that NT varies a little because of natural variability, and the measurements observed, however precise and reproducible the sonographer is, will exhibit a certain amount of variation around the reference value. Therefore, the differences between the two should approximately follow a normal distribution with mean µ0 = 0 and standard deviation sd. If for some reason, such as the arrival of a new sonographer with limited training in NT measurements and a tendency to overestimate NT, the measurements deviate from the expected value, the differences calculated will follow a normal distribution with mean µ1 = µ0 + δ and standard deviation sd. Let us consider the following null hypothesis, that the process is in control, with µ0 = 0, and the alternative hypothesis that the process is out of control, with µ1 = 0.125. In both cases the standard deviation is considered to be sd = 0.25 (empirical measurement). Because the data monitored follow a normal distribution, the sample weight is Wt = (Ot − Et)/sd − k, where Ot and Et are the observed and expected NT measurements respectively, and k the reference value. Therefore, the CUSUM score will deviate from zero only when the standardized differences are larger than k. Because the variable monitored is continuous and may deviate positively as well as negatively, two CUSUMs, one positive and one negative, are computed simultaneously. Here, the CUSUM is designed to detect a shift of half the standard deviation (µ1 − µ0 = sd/2) and so k = 0.25 (k = 0.5 g, with g the number of standard deviations to be detected) and the limits are set at h = ± 9.2 to optimize the properties of the test (see Appendix). Figure 1 shows the CUSUM score for 350 consecutive NT measurements at this center. The first 250 measurements are considered in control, with mean µ0 = 0; with the arrival of the second sonographer, the last 100 are out of control, with mean µ1 = 0.125. The CUSUM raises an alarm after the 264th measurement, namely only 14 procedures after the change in performance has occurred. Cumulative sums (CUSUM) score for 350 sequential nuchal translucency measurements. The performance was acceptable (the process was in control; the mean difference between the observed and expected measurements was nil) for the 250 first procedures (target) and unacceptable for the last 100 (the process was out of control; the mean difference between the observed and expected measurements was 0.125 mm). The CUSUM signaled after the 264th measurement that a change in performance had occurred. The CUSUM test has the ability to raise an alarm when poor performance occurs but it may also raise an alarm even though the level of performance is adequate. These alarms are called false alarms and can be regarded as being analogous to the Type I error rate (alpha level) of standard statistical tests. Usually, the properties of a CUSUM test are represented by its average run length under the null hypothesis (ARL0) and under the alternative hypothesis (ARL1). The ARL0 represents the number of procedures before a false alarm occurs when the performance is in control. Therefore, an alarm does not always indicate poor performance and an audit is always necessary to assess the performance after an alarm has been raised; for the present case, the mean ARL0 was 688, indicating that, even if the performance of the sonographers is adequate, a false alarm will be raised after approximately every 688 procedures. The ARL1 is the number of procedures observed before an alarm occurs, when the performance is indeed out of control; the ARL1 represents the responsiveness of the CUSUM test. For this study, the ARL1 was set at 34. Naturally, one should always try to minimize the ARL1 and maximize the ARL0, but because both vary in the same direction the researcher has to choose between responsiveness and the frequency of false alarms. Statistical process control methods should be implemented in a very simple manner. Ideally, the indicator monitored should be a marker agreed among the scientific community, with very little possibility for measurement errors or subjective interpretation, and it should not require any additional investigation to be made. The target and the properties of the tests should be defined beforehand. Data should be collected prospectively, analyzed independently, and meetings should be planned on a regular basis to discuss the results of monitoring; if needed, audits should be triggered. One last rule is that one should only monitor what one may be able to correct: the basis for monitoring is to improve performance, not solely to detect poor performance. The CUSUM test computes, at each time t, a score St defined by St = max(0, St−1 + Wt), with S0 = 0 and Wt the sample weight. The calculation of Wt depends on the endpoint monitored. For normally distributed data Wt = (Xt − µ0)/σ− k, where µ0 represents the mean of the process when considered as in control, σ the standard deviation of the series Xt (which is assumed to exhibit no serial correlation), and k is called the reference value. Based on the log-likelihood ratio, the CUSUM test parameter k is 0.5 g, where g represents the number of standard deviations one wants to detect. Usually, g is arbitrarily chosen between 0.5 and 2 in the medical literature, depending on the process monitored. The average run lengths under the null and alternative hypotheses are best determined by simulation, although some analytical approximations have been developed for Gaussian data. In practice, one should determine g first and compute different ARL0 and ARL1 values by varying h to obtain an acceptable compromise between a very responsive test (short ARL1) and too many false alarms (short ARL0). The CUSUM test for binomial data uses the Bernoulli distribution to determine the following Wt = log[(1 − p1)/(1 − p0)] if Xt = 0 (success), and Wt = log(p1/p0) if Xt = 1 (failure), with p0 and p1 the failure rates under the null and alternative hypotheses respectively. When monitoring endpoints such as counts of rare events, the Poisson distribution can be used if the assumption of normality is challenged. Using optimal weights based on the log-likelihood ratio leads to Wt = Xt − (λ1 − λ0)/(log(λ1)− log(λ0)), where λ0 and λ1 denote the mean of the Poisson distribution under the null and alternative hypotheses respectively.