Building factorial prompt matrices using query targets and configurable factor classes: A methods framework for conversational AI response behavior research

The term “factorial prompt matrix” (FPM) names and formalizes the emerging yet unstandardized practice of creating data structures that organize systematically varied prompts used to elicit responses from conversational AI (CAI) products. We introduce a methods framework for designing and implementing FPMs that, though broadly applicable, is primarily intended for social and behavioral scientists interested in controlled observational or experimental research on CAI response behavior. FPMs start from a principled sampling spine of CAI query “target” items that are then crossed with configurable factors drawn from four classes: (1) the large language model (LLM), (2) the product-system, (3) the user context, and (4) the query design. This produces a factorial grid of distinct prompts that can be passed to CAIs (e.g., using APIs). Using the ICD-11 (International Classification of Diseases, Eleventh Revision) as a spine for generating medical prompts, we demonstrate use of FPMs to analyze CAI information retrieval behaviors such as web search invocation. Key methodological contributions: A formalized data structure (FPMs) for systematically constructing and deploying prompts to CAIs. A modular, four-class schema for conceptually organizing the configurable factors of a CAI prompting protocol. Demonstration of FPMs across single-factor and multi-factor observational and experimental research designs.