A Cautionary Note Against Selective Applications of the Bayes Factor

Bayes-factor analysis becomes increasingly popular, among other reasons, because it allows to provide evidence for the null hypothesis which is not easily possible with the traditional frequentist approach. A conceivable strategy that apparently takes favorable aspects of both approaches on board, is to use traditional frequentist analyses first, and to support theoretically interesting nil effects by Bayesian analyses thereafter. Here we asked whether such a selective application of Bayesian analyses to only non-significant effects of foregoing frequentist analyses creates bias. In two simulation studies we observed that such selective application of Bayesian analyses in fact severely overestimates evidence in favor of the null hypotheses, when a true population effect exists. While this bias can be attenuated by using more informative priors in the Bayesian analyses, we recommend to not apply such selective combination of analytical approaches, but instead to use either frequentist or Bayesian analyses consistently.