Much Ado About Survey Tables: A Comparison of Chi-Square Tests and Software to Analyze Categorical Survey Data

Chi-square tests are often employed to examine the association of categorical variables, the homogeneity of proportions between two or more samples, and the goodness-of-fit for a specified distribution. To account for the complex design of survey data, variants of Chi-square tests as well as software packages that implement these tests have been developed. Nevertheless, from a survey practitioner’s perspective, there is a lack of applied literature that reviews and compares alternative options of survey Chi-square tests and their associated programming and output. This article aims to fill such a gap. Many modern statistical software packages for survey analysis are capable of computing either the Wald Chi-square test or the Rao-Scott Chi-square test, along with other types of Chi-square tests, including the Rao-Scott likelihood ratio Chi-square test and the Wald log-linear Chi-square test. This article focuses on these four types of Chi-square tests, and examines four statistical packages that compute them in SAS®, R, Python, and SUDAAN®. While the same type of tests using different packages yield similar results, different types of Chi-square tests may yield variations in p-values when conducting the same comparison. Sample programming code is included in Appendix for readers’ reference.