Compares code quality across multiple programming languages using various code generation tools, revealing key limitations.
Large Language Model (LLM)-based code generation tools have gained widespread adoption in software development. The code quality is an important concern that affects the practical usefulness of generated code suggestions. However, comprehensive evaluations of these tools’ code quality remain limited. In this paper, we conduct a systematic comparison of five mainstream code generation tools, including GitHub Copilot, Amazon CodeWhisperer, Tongyi Lingma, Tabnine, and Codeium, across five popular programming languages (C, C++, Java, JavaScript, Python). Using 2,613 programming problems from LeetCode, we generated and analyzed 65,325 code suggestions. We then evaluate their code quality through four dimensions: functional correctness, reliability, maintainability, and complexity. Our experimental results reveal substantial performance variations across tools, languages, and quality dimensions. The overall functional correctness remains limited, with even the best-performing tool achieving only moderate success rates. Reliability and maintainability present additional concerns, with bugs occurring more frequently in hard-difficulty tasks and code smells being pervasive across generated outputs. Although complexity is generally well-controlled, there are still outliers with excessive complexity across all tools. We believe that our study can provide valuable insights into the current state and limitations of LLM-based code generation technology.
No takes yet. Share an insight, caveat, or question.
Li et al. (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: