Statistical “Discoveries” and Effect-Size Estimation

Abstract Current methods of statistical inference are correct but incomplete. Small probability (α) of wrong null-hypothesis rejections can be misunderstood. Definitive rejections of null hypotheses, as well as interval assessments of effect sizes, are impossible in single cases with significance probabilities like .05. Sufficiently large sets of independent experiments and attained significance levels (p-values) should be registered. From such data it is possible to calculate least upper bounds for proportions of fallacies in sets of null-hypothesis rejections or effect-size assessments. A provisional rejection of a null hypothesis in a one-tailed test, or a one-sided confidence interval showing a nonzero effect, is here called a discovery. Consider a large number n of independent experiments with r discoveries. For r rejections of null hypotheses the proportion of fallacies Q has least upper bound Q max = (n/r — 1)α/(1 — α) < 1. For r confidence intervals, the proportion of fallacies is E = αn/r (assuming that no alternative is in the “wrong” direction).