Beyond Methodological Constraints: A Multilevel Framework for Evaluating Scientific Credibility

The current replication and credibility crises are often traced to flexibility in research methods and analyses, which can leave room for questionable research practices, inadvertent biases, and methodological and statistical errors. In response, reform efforts have emphasized stricter methodological and procedural constraints, such as preregistration and confirmatory designs. In this article, we develop an alternative, multilevel framework for evaluating scientific credibility. We examine factors that influence the credibility of research findings across levels of inference, including background assumptions, theory, methodology, and data. We argue that the credibility of results depends on interactions across these levels rather than adherence to a particular methodological approach. Our framework draws on two conceptions of evidential strength: logical and methodological severity. Logical severity concerns the strength of the inferential relation between theoretical claims and empirical evidence, whereas methodological severity concerns the capacity of research procedures to rule out error. We argue that, when relevant severity criteria are satisfied, credible results can be obtained through diverse methods, including both confirmatory and exploratory approaches. Our framework seeks to provide room for methodological pluralism while supporting core science reform goals, including credibility, replicability, and scientific informativeness.