Pairwise Multiple Comparisons: New Yardstick, New Results

Behavioral science researchers often wish to compare the means of several treatment conditions on a specific dependent measure. The author used a Monte Carlo study to compare familywise error controlling multiple comparison procedures (MCPs; Tukey, Bonferroni) with MCPs that were not developed to control the familywise error rate on the probability of correctly identifying the true underlying population mean configuration (true model rate). Recently proposed MCPs that are not intended to control the familywise error rate had consistently larger true model rates than did familywise error controlling MCPs. Furthermore, of the familywise error controlling MCPs investigated, the popular Tukey and Bonferroni MCPs had consistently lower true model rates than did other familywise error controlling MCPs.