Inter-unit reliability for quality measure testing

Facility-specific quality measures are commonly used to monitor dialysis facilities. To successfully develop, test and validate quality measures, a subset of facilities are often recruited for preliminary evaluations. To ensure that the facility-specific measures will achieve a desirable precision, it is often of interest to determine a minimum number of facilities that should be recruited. To achieve this, we propose a method based on the inter-unit reliability (IUR), which is commonly used to assess quality measures. Accurate estimates of the IUR are important to ensure that the quality measure will achieve a desirable precision. We first review existing methods of estimating the IUR for quality measures that are based on sample averages. We then generalize the IUR estimations to more complicated standardized measures. In particular, the confidence intervals of the IUR are calculated, with the width of this confidence interval measuring the precision of the estimate of the IUR. To assess the performance of the estimated IUR with various numbers of facilities, a simulation study is conducted. The IURs are then computed to develop and implement a quality measure that is used to guard against high ultrafiltration rates for adult dialysis patient with End-Stage Renal Disease. The estimated values are helpful to determine a minimum number of facilities that should be recruited in the measure testing process.