Answers are grounded in this paper and cite their sources.
To realistically evaluate a software testing or debugging technique, it must be run on defects and tests that are characteristic of those a developer would encounter in practice. For example, to determine the utility of a fault localization or automated program repair technique, it could be run on real defects from a bug tracking system, using real tests that are committed to the version control repository along with the fixes. Although such a methodology uses real tests, it may not use tests that are characteristic of the information a developer or tool would have in practice. The tests that a developer commits after fixing a defect may encode more information than was available to the developer when initially diagnosing the defect.
No takes yet. Share an insight, caveat, or question.
Just et al. (2018) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: