A Comparative Evaluation of ChatGPT-4o and DeepSeek-R1 for AI-Generated Code Quality Using Software Engineering Metrics

The paper presents an empirical evaluation of code quality generated by two of the most popular large language models, ChatGPT-4o and DeepSeek-R1, as an effort to evaluate their capabilities and weaknesses in real-world software development contexts. The study was conducted as a large-scale study consisting of an experiment that tested 600 problems in Java and Python programming, and the problems were chosen as those covering a wide range of software engineering problems. Each solution produced by AI was evaluated based on five primary components of the quality of the code: code structure and organization, method and class design, naming conventions, error handling and validation, and commenting and documentation. The choice of these metrics was to count not only the structural strength of code, but also the semantic one, when introducing an objective picture of code quality. The outcome indicates that DeepSeek-R1 is more likely to generate code with stronger maintainability-oriented quality, greater consistency, and better structural organization than ChatGPT-4o. The average scores were superior and the standard deviations were lower, and the failure rates were much lower (in all the quality parameters) in both languages in DeepSeek-R1. ChatGPT-4o scored high on the method design and naming conventions, but was more variable and performed less well on error handling and documentation, which are important to maintainability and robustness. The results indicate that DeepSeek-R1 achieved higher mean scores than ChatGPT-4o in code structure and organization for Python (8.89 vs. 7.49) and Java (8.83 vs. 7.07), method/class design for Python (8.73 vs. 7.11) and Java (8.65 vs. 7.06), error handling and validation for Python (7.15 vs. 5.06) and Java (7.15 vs. 4.69), and documentation/comments for Python (7.24 vs. 5.55) and Java (6.35 vs. 4.53). Functional correctness testing also showed higher full-problem correctness rates for DeepSeek-R1 in Java (77.8% vs. 68.4%) and Python (81.6% vs. 72.9%). ChatGPT-4o remained competitive in naming conventions, particularly in Java (7.44 vs. 6.61).