Collaboration between intelligent agents and large language models: A novel approach for enhancing code generation capability

Pre-trained Large Language Models (LLMs) have demonstrated significant potential in the Natural Language to Code (NL2Code) task. However, user-provided natural language descriptions are often ambiguous or misleading, resulting in poor code quality. Additionally, LLMs have limited ability to solve complex programming tasks with multiple requirements or higher difficulty levels. To address this, we developed a collaborative code generation framework integrating intelligent agents with LLMs. This framework optimizes code generation by dividing the NL2Code task into four stages: role definition, demand optimization, code writing, and code review. Based on the task-instruction-prompts and role-definition-prompts dataset we created, we fine-tuned a BART model to develop an intelligent agent enriched with programming knowledge. This agent generates more detailed prompts to guide LLMs at each stage, enhancing code generation capabilities. We evaluated our method on multiple datasets, including HumanEval, MBPP and LLMSecEval, using various metrics such as pass@k, logical error rate, code quality score, vulnerable@k, and secure@k to assess coding ability. To further strengthen our evaluation, we compared our approach against several specialized code-focused LLMs, including CodeGeeX, CodeLlama, and DeepSeek-Coder-V2. Our method improved the pass@1 metric by 18.9% and 23.2%, compared to the GPT-3.5 baseline and reduced the logical error rate from 38.2% and 29.1% to 19.3% and 13.6%. Moreover, the integration of the intelligent agent with GPT-4 led to notable performance improvements, with a 16.9% increase in pass@1, a 8.14% increase in code quality score, and a reduction in the logical error rate from 12.5% to 8.1%. Additionally, the Secure@1 metric for GPT-4 improved from 48.6% to 55.3%, reflecting enhanced code security. These experimental results demonstrate the efficacy of our method in advancing the code generation capabilities of LLMs.