Answers are grounded in this paper and cite their sources.
Context: Code review has become a core practice in collaborative software engineering, helping ensure code quality, detecting potential bugs, and supporting communication among developers. Prior research has shown that code review practices differ between production and test code, suggesting that established code review guidelines may fall short in the context of test and GUI-based test code. Particularly, GUI-based testing lacks adequate support during the code review process. To address this, we proposed a set of code review guidelines specifically designed for reviewing GUI-based test files. However, these guidelines have not yet been empirically evaluated, which limits their practical relevance. Objective: This study examines the extent to which code review comments on GUI-based tests align (explicitly or implicitly) with the concerns captured by the proposed guidelines and uses the findings to refine these guidelines. Method: To achieve this, we sampled code review comments discussing GUI-based test files across 100 open-source projects and manually analyzed 1000 pull requests to determine the extent to which the reviewers’ comments align with the proposed guidelines. Results: Review comments aligned with the proposed guidelines in 808 of 1000 pull requests. We found empirical support for 25 of the 33 guidelines in code review practice. The most frequently observed guideline concerns the correct use of testing techniques and exception handling, particularly regarding locators, explicit waits, and timeout behavior. Conclusion: The observed alignment suggests that the proposed guidelines capture concerns articulated in practice, indicating practical relevance for GUI-based test reviews. The results position the guidelines as practice-informed recommendations and motivate future work to evaluate their impact on review quality and outcomes.
No takes yet. Share an insight, caveat, or question.
Bauer et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: