The Meaning of Failed Replications: A Review and Proposal

Abstract The welcome rise of replication tests in economics has not been accompanied by a consensus standard for determining what constitutes a replication . A discrepant replication, in current usage of the term, can signal anything from an unremarkable disagreement over methods to scientific incompetence or misconduct. This paper proposes a standard for classifying one study as a replication of some other study. It is a standard that places the burden of proof on a study to demonstrate that it should have obtained identical results to the original, a conservative standard that is already used implicitly by many researchers. It contrasts this standard with decades of unsuccessful attempts to harmonize terminology, and argues that many prominent results described as replication tests should not be described as such. Adopting a conservative standard like this one can improve incentives for researchers, encouraging more and better replication tests.