Through the Looking Glass: Eye Tracking as a Cognitive Benchmark for Language-Model Semantics and Observer-Specific Semantic Alignment

Assessing semantic representations learned by language models remains challenging because conventional intrinsic benchmarks capture limited aspects of meaning and often correlate weakly with downstream performance. Eye tracking offers complementary, cognitively grounded evidence: gaze measures recorded during reading and scene viewing index online processing and attentional selection, although they should not be treated as readouts of semantic representations. This narrative review and methodological framework synthesizes research linking distributional and contextual language representations to eye-movement behavior, focusing on semantic relatedness, predictability, surprisal, model scale, layer-dependent alignment, and model–human correspondence. Evidence from naturalistic reading corpora shows that embedding-derived relatedness and language-model surprisal explain variance in fixation measures, while recent work demonstrates that cognitive alignment depends on model size, layer, tokenization, task, and measurement regime. We propose extending this approach from population-level model evaluation to observer-specific gaze-derived relational structure. For visual scenes, gaze transitions can be represented as candidate choices and relational graphs and compared with multiple semantic-relation matrices derived from language or vision–language models. Because gaze transitions are influenced by spatial distance, object size, salience, task demands, context, and preprocessing decisions, these factors must be controlled and, where causal isolation is intended, manipulated independently of semantic relations. We propose combining candidate-normalized hierarchical choice models, residualized gaze relations, representational similarity analysis with reliability and uncertainty assessment, and required independent behavioral validation to characterize convergence and divergence between model semantic geometry and observer-specific relational organization.