A Review on Code Generation with LLMs: Application and Evaluation

Code generation is a longstanding subject in the field of computer science and software engineering, which aims at realizing an agent capable of writing code automatically aligning with human desire. With the booming development of large language models (LLMs) in recent years, code generation techniques powered by LLMs with strong coding ability have caught many researchers' interest. In this study, we conduct a review of recent studies about code generation with LLMs, from the application of LLM-based code generation to the evaluation of LLM-generated code. We find, with the powerful code understanding and writing ability LLMs provide, these novel techniques can be applied to manage various software engineering tasks, and indeed boost the productivity of developers to a great extent. But we also find, as an equally important subject, the evaluation receives less attention from researchers than the application. We conclude some limitations in existing studies about the evaluation of code generated by LLMs, like inadequate quality characteristics considered. And we think more effort is needed to narrow the gap between research on the evaluation and the application.