What question did this study set out to answer?

This research aims to evaluate the potential of large language models in facilitating the co-evolution of grammar definitions and textual instances in domain-specific languages.

July 12, 2026Software & Systems Modeling0 citationsOpen Access

Leveraging LLMs to support co-evolution between definitions and instances of textual DSLs: a systematic evaluation

WZWeixing Zhang BJBowen Jiang FYFu Yuhong

Key Points

This research aims to evaluate the potential of large language models in facilitating the co-evolution of grammar definitions and textual instances in domain-specific languages.
Systematic evaluation of LLM-based solutions using Claude Sonnet 4.5 and GPT-5.2 across ten experimental runs per case.
Assessment focused on correctness of co-evolved instances and preservation of human-oriented information.
Analysis of performance metrics such as precision and recall, and response times across varying instance scales.
Achieved ≥ 94% precision and recall for instances with fewer than 20 lines needing modification.
Claude Sonnet 4.5 maintained 85% recall at 40 lines, while GPT-5.2 struggled with larger instances.
Performance impacted by grammar evolution complexity and deletion granularity, with limited prompt transferability noted.

Abstract

Abstract Software languages evolve over time for various reasons, such as the addition of new features. When the language’s grammar definition evolves, textual instances that originally conformed to the grammar become outdated. For DSLs in a model-driven engineering context, there exists a plethora of techniques to co-evolve models with the evolving metamodel. However, these techniques are not geared to support DSLs with a textual grammar—applying them to textual language definitions and instances may lead to the loss of information from the original instances, such as layout information and comments, which are valuable for software comprehension and maintenance. This study systematically evaluates the potential of Large Language Model (LLM) -based solutions in achieving grammar and instance co-evolution for textual DSLs. By applying two advanced language models, Claude Sonnet 4. 5 and GPT-5. 2, and conducting ten experimental runs per case across ten case languages, we evaluate both the correctness of co-evolved instances and the preservation of human-oriented information such as comments and layout. Our results indicate high performance on small-scale cases (≥ 94% precision and recall for instances with fewer than 20 lines requiring modification), but performance degraded with scale: Claude Sonnet 4. 5 maintained 85% recall at 40 lines while GPT-5. 2 showed greater sensitivity, failing entirely on the two largest instances. Instance scale also substantially impacts processing efficiency, with Claude’s response time increasing nearly 18-fold for the largest case. In addition, we observe that grammar evolution complexity and deletion granularity impact performance more than change type alone, and prompt transferability across LLMs is limited. These findings identify the conditions under which LLM-based co-evolution succeeds and its scalability limitations, providing practitioners with insights into applicability and researchers with directions for addressing current limitations.

Ask AI

Helpful

Bookmark

View Full Paper