Analyzing Binary Judgments: An Empirical and Simulation-Based Comparison of Aggregated and Trial-Level Approaches for Hypothesis Testing

Binary judgments are widely used in psychological research to investigate cognitive processes and decision making. Statistical approaches for analyzing such data differ in whether responses are aggregated (e.g., analysis of variance, ANOVA) or modeled at the trial level (e.g., generalized linear mixed models, GLMMs), and in whether results are interpreted as choice proportions or in terms of signal detection theory (SDT) parameters for response bias and discrimination ability. We examined how these approaches differ in specification, interpretation, and results. First, we reanalyzed 20 openly available datasets on the illusory truth effect (212,555 observations), comparing by-subjects (F1) and by-items (F2) ANOVAs of choice proportions, their combination via min-F', a two-step SDT approach (ANOVA of individual SDT estimates), and seven GLMMs with random-effects structures of varying complexity. Second, a Monte Carlo simulation assessed Type I error rates, power, and parameter recovery. GLMMs with random slopes for both participants and items controlled Type I error rates best but often resulted in singular fits. GLMMs with random slopes only for participants were favored by AIC and BIC, but inflated Type I error rates for interactions by ignoring item-specific variability in the repetition effect. Random-intercept-only GLMMs, F1 and F2 ANOVA, and two-step SDT were too liberal for some effects, whereas min-F' was overly conservative. Unlike aggregated analyses, GLMMs account for both participant and item variability and estimate SDT parameters with less bias. We therefore recommend GLMMs with theoretically justified random-effects structure, simplified only if necessary and with the resulting risk of Type I errors for confirmatory hypothesis testing.