Talking Like the Judge: Linguistic Style Matching as a Mechanism of Evaluation Reactivity in Iterative LLM-as-Judge Pipelines

This working paper investigates evaluation reactivity in iterative LLM-as-judge pipelines, where a language model generates a response, receives an evaluation score from another model, and revises its output to improve future scores. The study introduces a psycholinguistic approach to analyzing evaluation dynamics through Language Style Matching (LSM), a measure of function-word similarity originally developed in human communication research. Using a fully open and reproducible implementation of LSM, the paper examines whether generator models gradually converge toward the linguistic style of the judging model during repeated evaluation-revision cycles. Contrary to the original hypothesis, the results show that linguistic style matching decreases rather than increases across revision cycles, suggesting that optimization pressure may drive stylistic divergence from the evaluator's language. The study also finds a significant negative relationship between content-word overlap with evaluation-rubric language and judge scores, indicating potential design risks in evaluation prompts and rubrics. In addition, the paper documents an unplanned but highly reproducible failure mode in score-only revision pipelines. When models receive only scalar evaluation scores without access to the original task or prior response, iterative improvement collapses into persistent clarification-seeking behavior, accompanied by a substantial and sustained drop in evaluation scores. The work contributes to the growing literature on LLM evaluation, measurement validity, reward optimization, and AI feedback loops by introducing a linguistic perspective on evaluation reactivity and providing openly reproducible methods, code, and metrics for future research. Keywords: Large Language Models, LLM-as-Judge, Evaluation Reactivity, Language Style Matching, Psycholinguistics, RLHF, AI Evaluation, Measurement Validity, Reward Hacking, Agentic Systems.