Mini-Many Analysts: Using Registered Reports to Learn about Analytical Flexibility

Analytical flexibility describes the multiple defensible data analytic options researchers can pursue in a given study. Analytical flexibility can include practices like applying different methods for data cleaning, exclusion criteria, and statistical tests. Analytical flexibility is increasingly important for students to learn about because it impacts low replicability and credibility. Registered reports reduce analytical flexibility by submitting a manuscript to a journal in two stages, where journals provide an in-principle acceptance after Stage 1, committing to publishing the results. We describe and demonstrate an activity for psychology undergraduate courses to explore analytical flexibility within a registered report. Students read a first-stage registered report, received the data, and implemented the analysis plan as described. Analysts generate a product summarizing their results, and compare with the final Stage 2 and other analysts. This activity helps students to achieve multiple important learning outcomes focused on the impact of analytical flexibility on research conclusions, advantages and limitations of open science practice, and implementation of statistical methods in a real data context. This type of project could be an excellent option for a final project in an advanced undergraduate research methods or statistics course.