Conditional Permutation Tests and the Propensity Score in Observational Studies

Abstract In observational studies, the distribution of treatment assignments is unknown, and therefore randomization tests are not generally applicable. However, permutation tests that condition on sample information about the treatment assignment mechanism can be applicable in observational studies, provided treatment assignment is strongly ignorable. These tests use the conditional distribution of the treatment assignments given a sufficient statistic for the unknown parameter of the propensity score. Several tests that are commonly used in observational studies are particular instances of this general procedure. Moreover, conditional permutation tests and covariance adjustment are closely related, in the sense that the conditional permutation distribution of the covariance adjusted difference leads to the same inferences as the conditional permutation distribution of the unadjusted difference of sample means. A backtrack algorithm is developed to permit efficient calculation of the exact conditional significance level, and two approximations are discussed. A clinical study of treatments for lung cancer is used to illustrate the technique. Conditional permutation tests extend previous large sample results on the propensity score by providing a basis for exact tests and confidence intervals in small observational studies when treatment assignment is strongly ignorable.