Evaluates GitHub Copilot’s multiple solutions in coding problems, suggesting how many should be checked for accuracy.
GitHub Copilot is a generative artificial intelligence (GenAI) tool designed to support software developers when writing source code. As such, GitHub Copilot provides a feature of generating up to ten solutions to a given task, enabling developers to choose a suitable option. With this study, we provide recommendations on how to reasonably balance finding and checking solutions. We evaluate the multi-suggestion feature using 2025 LeetCode coding problems for which we generated a total of 17 048 solutions using GitHub Copilot in Python. The recommended approach varies by coding problem characteristics. However, generally, it is enough to check only a few solutions to find a correct one (for easy coding problems) or be certain that there is none (for more difficult problems).
No takes yet. Share an insight, caveat, or question.
Oertel et al. (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: