Answers are grounded in this paper and cite their sources.
Context Replication plays a critical role in building cumulative scientific knowledge. However, in the context of Empirical Software Engineering (ESE), replication efforts face persistent difficulties, both technical and methodological, which hinder reproducibility and generalizability. We explored whether the problems were due to formal issues, such as under‐specification or miscommunication, or intrinsic reasons, i.e., the existing replication procedures may not meet the researchers’ needs. Objective To understand how ESE researchers conduct experiments in real‐world settings. Method We conducted an ethnographic study with an experimental software engineering group, using interviews, observations, and analysis of internal documentation. Results We have created conceptual and process models representing experimentation, replication, and synthesis in the target research group. These models align with mainstream procedures at a high level but often break down in practice during specialized tasks such as coordination or documentation. The experimental process observed in the group differs from common assumptions and textbook descriptions in terms of (1) the number and diversity of activities involved, (2) the presence of differentiated roles, (3) the granularity of conceptual elements, and (4) the varying perspectives across subareas or families of experiments. Conclusions The discrepancies between actual laboratory practices and idealized process models may hinder knowledge transfer and complicate replication. We plan to extend this research by involving additional research groups, for instance, through surveys, to examine the generalizability of our findings.
No takes yet. Share an insight, caveat, or question.
C. et al. (2025) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: