Comparison of tests for categorical data from a stratified cluster randomized trial

Two features commonly exhibited by randomized trials of health promotion interventions are cluster randomization and stratification. Ignoring correlations between individuals within clusters can lead to an inflated type I error rate and hence a P-value which overstates the significance of the result. This paper compares several methods for analysing categorical data from a stratified cluster randomized trial. We propose an extension of a method from survey sampling that uses the design effect to reduce the effective sample size. We compare this with three methods from Zhang and Boos that extend the standard Cochran-Mantel-Haenszel (CMH) statistic by using appropriate covariance matrices, and with a bootstrap method. The comparison is based on empirical type I error rates from a simulation study, in which the number of clusters randomized is small, as in most public health intervention studies. The method that performs consistently well is one of the Zhang and Boos extensions of the standard CMH statistic.