Researchers’ Perceptions of Statistical Significance Contribute to Bias in Health and Exercise Science

We surveyed researchers in the health and exercise sciences to explore different areas and magnitudes of bias in researchers’ decision making. Participants were presented with scenarios (testing a central hypothesis with p = .06 or p = .04) in a random order and surveyed about what they would do in each scenario. Participants showed significant bias in deciding to randomize additional participants when p = .06 (24.5%) compared to when p = .04 (6.5%), χ2 = 19.86, odds ratio (OR) = 13.5. Based on this bias to randomize additional participants, we present statistical simulations showing the cumulative negative effect this bias has on the false positive rate for a field. While it is important for researchers to have flexibility in their data-processing decisions, there is a cost/benefit trade-off to experimenter degrees of freedom and it is important that these decisions be unbiased. These data highlight the importance of a priori analysis plans in research.