Simulation-Based Comparison of Bayesian Inference Methods in Hierarchical Bayesian Computational Models

When rejecting and accepting hypotheses, it is important to determine the risk of false positive conclusions. Hierarchical Bayesian models are frequently applied in cognitive science, psychology, cognitive neuroscience, and psychiatry to reduce the variability of low-level parameters through partial pooling, and to obtain more robust parameter estimates for complex models using limited data. This may lower the risk of false positive inferences from lower-level estimates. In contrast, higher-level (e.g. group-level) estimates may be more affected by modeling choices (e.g. prior distributions), population-level variances in effects, and inference methods, which may in turn affect false positive rates for group-level effects. Here we used simulation and computational modeling to systematically examine false positive rates and effects of specific modeling choices on inferences in a simple hierarchical Bayesian temporal discounting model. To this end, 600 data sets of a temporal discounting task with two conditions were simulated, drawing individual condition effect parameters from a null distribution with varying standard deviations. For each simulated data set, the group-level mean change in the discount rate was modeled with eight different prior variances of the condition effect. Results were subsequently evaluated using several commonly applied Bayesian inference methods: the Savage-Dickey Bayes factor (BF), the directional Bayes factor (dBF), P(effect 0), and the highest density interval (HDI) against zero decision rule. Results confirm that inference methods that directly depend on the posterior distribution can be used regardless of prior variances, whereas the Savage-Dickey BF should only be used with sufficiently informative priors. Based on the simulation work, decision thresholds that ensure false positive rates below 5% are provided for each inference method. Results are discussed in the context of computational modeling approaches in cognitive (neuro-)science and psychology.