A Grounded Theory Study to Guide AI‐Driven Code Comment Improvement

Current automated code summarization research focuses primarily on mimicking existing human‐written comments, ignoring the fact that many baseline comments are incomplete or misaligned with developer needs. This paper presents a novel approach to intentionally improve code comments along different quality axes by rewriting those comments with customized artificial intelligence (AI)‐based tools. We first conduct an empirical study with developers, followed by grounded theory to develop a theoretical construct we define as comment refocusing. This theory identifies seven distinct quality axes grouped into two dimensions: Internalization (anchoring comments in implementation logic) and Externalization (projecting comments into system context). Based on this theoretical foundation, we propose a two‐stage alignment procedure using reinforcement learning from AI feedback (RLAIF) followed by human feedback (RLHF) to steer large language models toward these quality axes. We implement this procedure using a large‐scale teacher model (GPT‐4o) and distill the knowledge into smaller, privacy‐preserving models (e.g., jam, CodeLlama‐13B) capable of in‐house deployment. Our evaluation demonstrates that our procedure improves code comments along the quality axes, satisfying specific developer‐defined quality goals. We release all data and source code in an online repository for reproducibility.