Funnel plots for comparing provider performance based on patient-reported outcome measures

BACKGROUND: Patient-reported outcome measures (PROMs) often produce skewed distributions of individual scores after a healthcare intervention. For health performance indicators derived from skewed distributions, funnel plots designed with symmetric control limits may increase the risk of false alarms about poor performance. AIM: To investigate the accuracy of funnel plots with symmetric control limits when comparing provider performance based on PROMs. METHODS: The authors used a database containing condition-specific PROMs for 17,453 hip replacements and 7656 varicose vein procedures performed by providers in the English NHS. The mean postoperative PROM score, adjusted for patient characteristics, was used as the measure of performance. To compare performance, symmetric 99.8% control limits were calculated on funnel plots, 3 SDs away from the overall mean on either side. These were compared to control limits derived directly from percentiles of simulated (bootstrap) distributions of mean scores. RESULTS: The simulated control limits on funnel plots for both procedures were asymmetric. The empirical probability of falling outside the symmetric 99.8% 'poor performance' control limit was inflated from the stipulated rate of 0.1% to 0.2-0.3% for provider sample sizes of up to 150 procedures. The authors observed that, out of 237 providers of hip replacement, eight had adjusted mean scores that exceeded the symmetric 'poor performance' limit compared with only five that exceeded the corresponding simulated limit. In other words, three (1.3%) were differently classified. For varicose vein surgery, five out of 160 providers exceeded the symmetric limit and four exceeded the simulated limit, that is, 1 (0.6%) was differently classified. CONCLUSIONS: When designing funnel plots for comparisons of provider performance based on highly skewed data, the use of simulated control limits should be considered.