What does this research mean for the field?

While the semantic decisions of large language models remain generally stable under instruction paraphrasing, their reliability in adhering to structured output constraints varies substantially depending on both the paraphrase and the model. Novelty: ClaimNovelty.INCREMENTAL. Consensus alignment: ConsensusAlignment.NEUTRAL.

What question did this study set out to answer?

To determine how semantically equivalent prompt paraphrases impact the reliability with which large language models follow strict structural and formatting constraints.

September 8, 2026Open Access

Probing Instruction Execution Stability of Large Language Models under Semantic Paraphrase

View Full Paper

Ask AI

Bookmark

Authors

BMBodhisatta Maiti DCDebshree Chowdhury

Discussion

Member takes

Implication

Controlled evaluation reveals format adherence varies despite stable semantic decisions in eight instruction-tuned language models, highlighting that execution robustness is independent of accuracy.

Key Points

To determine how semantically equivalent prompt paraphrases impact the reliability with which large language models follow strict structural and formatting constraints.
Tested eight open-source instruction-tuned large language models on a classification task bound by a strict JSON output contract.
Evaluated multiple instruction-preserving paraphrases across repeated runs to distinguish paraphrase sensitivity from inherent stochastic variation.
Differentiated evaluation metrics between semantic agreement (label output consistency) and execution agreement (format constraint adherence).
Semantic classification decisions remain generally stable across varied instruction paraphrases.
Structured output format adherence fluctuates substantially depending on the specific paraphrase phrasing and model architecture.
Execution robustness represents a distinct, model-dependent capability that is not captured by conventional task accuracy metrics.

Cite This Study

Maiti et al. (2026) studied this question.

synapsesocial.com/papers/6a9fd70a58e84d0ff5b458c3 https://doi.org/10.5281/zenodo.19102312

View Full Paper

Ask AI

Bookmark

Implication

Controlled evaluation reveals format adherence varies despite stable semantic decisions in eight instruction-tuned language models, highlighting that execution robustness is independent of accuracy.