SF0004: Measurement Instruments and Validation Protocols Measurement Instruments and Validation Protocols (SF0004) defines the Multi-Turn Coherence Scale, Revised (MTCS-R), a behavioral measurement instrument for evaluating longitudinal coherence in extended human-AI interaction. MTCS-R operationalizes relational coherence as an observable property of interaction trajectories rather than individual responses, addressing a documented gap in dialogue evaluation methodology where standard metrics overwhelmingly operate at the turn level, treat coherence as unidimensional, and evaluate AI output in isolation rather than the joint conversational system. The instrument evaluates five analytically separable dimensions of cross-turn stability: D1 Thematic Persistence: maintenance of intended topic trajectory across turns D2 Contextual Integration Depth: incorporation and synthesis of earlier content in later turns D3 Register Homeostasis: stability and appropriateness of communicative style D4 Meta-Conversational Structuring: observable maintenance and organization of the interaction itself D5 Epistemic Calibration: consistency and appropriateness of certainty, uncertainty, and knowledge-boundary marking Each dimension is defined by behaviorally anchored descriptors on a 1 to 5 rating scale designed for trained human raters. The five-dimension structure reflects a principled decomposition based on classes of coherence degradation that can occur independently of one another; each dimension corresponds to a distinct failure mode that existing unidimensional metrics collapse into a single score. The deposit includes seven files: the SF0004 paper (PDF, with editable Word and Markdown forms available at the canonical record), the MTCS-R Rater Training Packet (annotated example transcripts, dimension definitions, calibration discussion guide), the MTCS-R Scoring Template (Excel workbook with scoring forms and simple agreement calculation), the MTCS-R Code Examples archive (Python scripts for semantic drift and perplexity analysis, R script for ICC and Krippendorff's alpha computation, sample data), and three deposit-support documents (READMEFIRST, MANIFEST, LICENSE). Epistemic status: MTCS-R is a measurement instrument specification, not an empirically validated instrument. The behavioral anchors and the proposed five-factor structure are theoretically motivated and practitioner-refined, not empirically confirmed. The paper defines what to measure and how to score it; the validation pathway (Sections 9 and 10 of the paper) specifies the staged reliability and factor-structure studies required to establish that it measures what it claims to measure. Scores produced prior to Phase 1 reliability testing should be interpreted as preliminary and exploratory. If Phase 2 factor analysis does not confirm the five-factor structure, the anchors, training materials, scoring template, and reporting format would require substantive revision rather than incremental adjustment. Document ID: SF0004 Version: 3. 4. 1 Author: Thomas W. Gantz Affiliation: Synthience Institute License: CC-BY 4. 0 For published work and Institute information: synthience. org
Thomas Gantz (Wed,) studied this question.