Answers are grounded in this paper and cite their sources.
We present a benchmark dataset for evaluating method-level code generation task. The benchmark contains a dataset of 175 samples for automated evaluation and a dataset of 161 samples for manual evaluation. We also present a new metric for automatically evaluating the correctness of the generated code, and a set of criteria to manually evaluating the overall quality of the generated code.
No takes yet. Share an insight, caveat, or question.
Hao et al. (2022) studied this question.