Efficiency Evaluation Framework for LLM-Generated Code

The impressive capabilities of large language models (LLMs) in natural language processing have led to their increasing use for code generation. Comprehensive evaluation of LLM‑generated code remains, however, an open challenge. While correctness is well‑studied, another crucial aspect of code quality, efficiency, is still to be explored. This paper presents a new framework for assessing the efficiency of generated code based on the Mercury and EvalPerf benchmarks. We use additional test case generation and sandbox improvements to refactor Mercury’s pipeline, integrating three types of resource measurements (runtime, CPU instruction count, and peak memory usage) and three code quality metrics (Pass@1, Beyond, and DPS). Our evaluation of six LLMs (Phi‑1, Phi‑2, Code Llama, QwQ‑32B, Qwen3, and DeepSeek‑V3) shows that large LLMs vastly outperform smaller models and generated solutions are usually better optimized for time than memory. Analyzing measurement and metric variance, we verify the stability of our approach. Our framework is extensible, providing a foundation for future code quality research.