The Focal v Control Test: Dealing With P‐hacking Concerns in Meta‐analytical Research

While overwhelming consensus exists that p‐hacking practices trouble management literature, the issue remains largely undiscussed in meta‐analytical research. This is surprising, since meta‐analyses traditionally deal with these types of systematic biases. Instead, meta‐analytical research in the field of management treats systematic bias coming forth from p‐hacking as a form of publication bias, to be solved via existing publication bias tests. In this paper, we argue that p‐hacking creates a unique type of bias which cannot be detected and dealt with by the currently employed publication bias tests—which have not been created for doing so. We introduce the ‘focal v control test’, as a simple diagnostic tool that exploits differences in researchers' incentives to p‐hack. We validate the approach in three ways: by reproducing the z‐statistic distribution of proof of p‐hacking in focal variables, for the control relationships in the same set of papers, by applying the test to a published meta‐analysis and through a simulation study. The test detects p‐hacking‐induced bias that conventional methods miss, particularly in fields reliant on large archival datasets and institutionalized sets of controls.