PROMPTECHO: A Pilot Study of Output Drift Under Developer-Style Prompt Variation

Studies of prompt sensitivity usually rewrite benchmark prompts with templates that the researchers designed. Developers rewrite prompts differently. They add background, split a request into steps, tighten a constraint, or ask for another output format. This paper is a small pilot on what those edits do to a model’s output. I define six categories of developer-style edits, build 24 pairs of prompts that ask for the same thing in two phrasings (12 code, 12 reasoning), and score each pair with a drift measure called PES, which runs from 0 (same behavior) to 1. For code, PES is the share of test inputs on which the two generated functions disagree. For reasoning, it compares final answers. Across 20 scored pairs, Claude Sonnet 5.5 averaged 0.063 and Phi-3-mini 0.131, and the interval for that gap includes zero. Nearly all of the drift came from pairs that add a constraint and from one pair on which Phi-3-mini answered the wrong question. Edits that only added context, steps, politeness, format instructions or a rephrasing produced no drift for Claude and one failure for Phi-3-mini. The sample is small and most pairs are authorwritten, so I treat the results as exploratory.GitHub Repository: https://github.com/Shriniketan369/PromptEcho