The Case for Magnitude-based Inference

Dear Editor-in-Chief, We respond here to a critique of magnitude-based inference (MBI) by Welsh and Knight (8). They asserted that MBI is just another form of null hypothesis significance testing (NHST), whereas it is philosophically and statistically distinct. Type 1 and Type 2 errors of MBI are conceptually quite unlike those of NHST. MBI forces researchers to define and justify clinically, practically, or mechanistically meaningful values of an effect. MBI then provides a framework for interpreting uncertainty in the effect in relation to these values, making it attractive to end users. MBI is actually a “reference Bayes” inferential method, combining the usual confidence interval with a noninformative prior belief (7). Noninformative prior beliefs are justified because prior knowledge is often vague (3). Moreover, MBI has the practical advantage of using standard sampling theory to obtain an intuitive (Bayesian) interpretation of the confidence interval as the likely range for the true effect (2,3,7). As such, MBI is quite possibly the ideal frequentist–Bayesian hybrid. Utilizing the logical fallacy of the straw man, Welsh and Knight (8) implied that MBI ignores data structure, multiple covariates, distribution and scale of the outcome variable, and presentation of effect size—all issues that we have attended to. They also practiced “mathematistry”—statistical theorizing that redefines a real-world problem (here, making robust inferences) without solving it (1,5). Using analytical formulas and simulation, we have verified that MBI Type 1 error rates (false discoveries of clear substantial effects when the true effect is null) are much less than Welsh and Knight (8) presented. The rates are acceptable given that the “errors” are accompanied by probabilistic terms representing level of evidence. For example, in a controlled trial such as the one analyzed by Welsh and Knight (8) (typical error of three times the smallest important effect), a true null produces clear possibly substantial effects only 21% of the time with nonclinical MBI, whereas the rates for likely substantial, very likely substantial, and most likely substantial are only 14%, 1.6%, and 0.1%, respectively. With conservative clinical MBI, the corresponding rates are only 0.0%, 2.1%, 0.9%, and 0.1%. Publication bias will decline if the criterion for manuscript acceptance is clear rather than significant, tempering any concerns about overall Type 1 error rates with MBI. In our simulations of the above controlled trial, trivial true effects produce trivial to small clear effects on average, but significant effects (P < 0.05 or P < 0.01) are mostly small to moderate. The differences arise from gratifyingly higher rates of potentially publishable clear effects with MBI (36%–81%) than of significant effects with NHST (1%–11%). The sample size spreadsheet at Sportscience (sportsci.org) produces correct estimates for NHST (verified by G*Power 3 software [4]) and MBI (verified by the Type 1 and Type 2 error rates shown). In conclusion, Welsh and Knight (8) proposed that we either present confidence intervals or use “a fully Bayesian analysis” in place of MBI. However, they were silent on how to make inferences with confidence intervals, and full Bayesian analyses are not about to become universal: “we will all be Bayesians in 2020” (6) seems most unlikely. Meanwhile, the robust hybrid approach of MBI is genuinely progressive and becoming popular.