What question did this study set out to answer?

To examine the distinct sources of replication failure in scientific practice, highlighting the role of causal architecture.

March 27, 2026Open Access

Beyond False Positives: The Replication Crisis and Architectural Realism

Key Points

To examine the distinct sources of replication failure in scientific practice, highlighting the role of causal architecture.
Analyzed existing literature on the replication crisis and its critiques.
Critiqued the standard diagnosis of replication failures and suggested a nuanced perspective.
Illustrated arguments with examples from cognitive and social psychology, as well as specific experimental setups.
Identified two distinct sources of replication failure: false positives and locally real findings.
Argued that standard reforms primarily address false positives but not the issue of non-transferability.
Emphasized the need for theoretical work on causal architecture to truly understand replication.

Abstract

The replication crisis is standardly diagnosed as a problem of scientific practice. Underpowered studies, flexible analyses, and misaligned incentives produce false positives that fail to replicate. The reform agenda that has followed---pre-registration, open data, larger samples, and incentive reform--is well-targeted at that diagnosis. This paper argues, however, that the diagnosis is incomplete. Replication failure has two distinct sources. The first is the false positive: a finding that was never real and is correctly identified as spurious when it fails to replicate. The second is a locally real finding within a specific material configuration that did not transfer to the replication setting. Underlying this conflation is a tacit assumption that nomothetic science discovers stable, context-independent causal structure--an assumption contested in this paper by treating the nomothetic/idiographic distinction not as a binary but as a continuum of generalisation. Drawing on architectural realism--a development of Cartwright's nomological machine framework--the paper argues that these two sources of failure are empirically indistinguishable without theoretical purchase on the complete configuration of all causal components. The reform agenda addresses the first source but cannot address the second, because non-transferability is not a defect of the original study. Three examples illustrate the argument: the differential replication rates of cognitive versus social psychology, the zebrafish novel tank test as an explicit attempt to engineer transferability, and the Large Hadron Collider as the limiting case of apparatus specification. The paper concludes that replication, properly understood, is a test of whether two settings share the causal architecture required to sustain a pathway to an outcome--not a test of whether a field-level truth was correctly detected. Closing the gap between these two conceptions requires theoretical work on causal architecture that the reform agenda does not supply.

Read Full Paperexternally

Mark Helpful

Bookmark

Relay

View Full Paper