Error statistics, Bayes-factor Tests and the Fallacy of Non-exhaustive Alternatives

In this paper I discuss a fundamental contrast between two types of statistical tests now in use: those where the post-data inferential assessment is sensitive to the method’s error probabilities—error statistical methods (e.g., statistical significance tests), and those where it is insensitive (e.g., Bayes factors). It might be thought that if a method is insensitive to error probabilities that it escapes the inferential consequences of inflated error rates due to biasing selection effects. I will argue that this is not the case. I discuss a recent paper advocating subjective Bayes factors (BFs) by van Dongen, Sprenger, and Wagenmakers (VSW 2022). VSW claim that the comparatively more likely hypothesis H passes a stringent test, despite insensitivity to the error statistical properties of that test. I argue that the BF test rule they advocate can accord strong evidence to a claim H, even though little has been done to rule out H’s flaws. There are two reasons the BF test fails to satisfy the minimal requirement for stringency: its insensitivity to biasing selection effects, and the fact that H and its competitor need not exhaust the space of possibilities. Data can be much more probable under hypothesis H than under a chosen non-exhaustive competitor H’, even though H is poorly warranted. I will recommend VSW supplement their BF tests with a report of how severely H has passed, in the frequentist error statistical sense. I begin by responding to the criticisms VSW raise for a severe testing reformulation of statistical significance tests. A post-data severity concept can supply a transparent way for skeptical consumers, who are not steeped in technical machinery, to check if errors and biases are avoided in specific inferences that affect them.