Generative operationalism: when LLM-based item generation displaces construct theory in educational and psychological measurement

Large language models (LLMs) are rapidly changing automated item generation (AIG) by enabling researchers to produce extensive pools of educational and psychological items with little time and human labor. Although this development is usually discussed as a technical improvement in test construction, it raises a more fundamental philosophical question: What warrants the claim that a machine-generated item measures the attribute assigned to it? This paper introduces the concept of generative operationalism to describe a mode of test development in which an ordinary-language construct label is supplied to a generative model, its outputs are treated as operationalizations of that construct, and post-hoc psychometric regularities are subsequently interpreted as evidence that the construct has been successfully measured. Drawing primarily on causal realism, we argue that LLM-generated items are not necessarily invalid for measuring their target attributes, but rather, theory-displacing uses of AIG weaken the epistemic warrant for the proposed causal pathway from the target attribute to the observed response. They may also produce semantic redundancy, construct underrepresentation, method variance, and residual dependence that generate apparently desirable psychometric properties without establishing that the intended attribute causally influences responses. The central problem is therefore not machine authorship itself, but the substitution of linguistic plausibility and statistical regularity for an independently articulated theory of the construct and its response processes. We conclude by specifying conditions under which LLMs can function as theory-implementing instruments rather than engines of generative operationalism .