Statistical Reporting Inconsistencies in Experimental Linguistics

The present paper investigates the prevalence of statistical reporting consistencies across articles in eight experimental linguistics journals published between 2000 and 2023. Using the R package “Statcheck”, we retrieved 39532 statistical test and assessed whether p-values were consistent with their test statistic and degrees of freedom. Half of all articles that used null hypothesis significance testing contained at least one inconsistent p-value. One in eight articles contained an inconsistency that may have affected the statistical conclusion. The inconsistency rates were comparable across journals and seem stable over publication years. We discuss possible reasons for this high rate and offer actionable steps for authors, reviewers, and editors to remedy this state-of-affairs.