Analysis and sample-size determination for 2 K audit experiments with binary response and application to identification of effect of racial discrimination on access to justice

Abstract Social scientists have increasingly turned to audit experiments to investigate discrimination in the market for jobs, loans, housing and other opportunities. In a typical audit experiment, researchers assign ‘signals’ (the treatment) to subjects at random and compare success rates across treatment conditions. In the recent past there has been increased interest in using randomized multifactor designs for audit experiments, popularly called factorial experiments, in which combinations of multiple signals are assigned to subjects. Although social scientists have manipulated multiple factors like race, gender and income, the analyses have been mostly exploratory in nature. In this paper we lay out a comprehensive methodology for design and analysis of 2K factorial designs with binary response using model-free, randomization-based Neymanian inference and demonstrate its application by analysing the audit experiment reported in Getting a Lawyer While Black (Libgober, 2020). Specifically, we integrate and substantially extend several sections of the randomization-based, finite-population literature for binary outcomes, including sample size and power calculations, and nonlinear factorial estimators.