What question did this study set out to answer?

January 6, 2026Open Access

Layer-wise Embedding-Output Dynamics Across LLM Families: Evidence for Phase-Structured Decision Commitment

Key Points

This research aims to explore the relationship between embedding geometry and output preferences across various large language model families.
Analyzed four LLM families: Pythia-6.9B, Llama-3.1-8B, Apertus-8B, and Gemma-2B.
Employed a pair-level centroid-asymmetry metric computed across 230 statement pairs with 10,000 bootstrap resamples.
Introduced a new bootstrapping framework to address artifacts of Simpson's Paradox in previous analyses.
Found phase-structured dynamics in embedding-output relationships.
Documented that decision commitment depth varies by architecture, with Llama committing early and Pythia committing late.
Identified distinct processing regimes associated with chat templates in instruction-tuned models.

Abstract

We investigate the layer-wise relationship between embedding geometry and output preferences across four diverse LLM families: Pythia-6.9B, Llama-3.1-8B, Apertus-8B (multilingual), and Gemma-2B (SFT). Using a pair-level centroid-asymmetry metric computed across 230 statement pairs with 10,000 bootstrap resamples, we find that embedding-output dynamics are phase-structured: the correlation between embedding clustering and output preference changes sign systematically across network depth. Key Findings:• Late-layer inversion is architecturally robust in larger base models (confirmed in 3/4 tested families).• Decision commitment depth is architecture-dependent (e.g., Llama commits early, Pythia commits late).• Chat templates are associated with a distinct processing regime in instruction-tuned models.• Small SFT models (Gemma-2B) represent a boundary condition where phase structure does not reliably emerge. Methodological Note:This study introduces a pair-level bootstrapping framework (n=230) that resolves Simpson's Paradox artifacts observed in previous category-level aggregations. Related Work:This work continues the research line established in D'Elia (2025): Uniformity Asymmetry (DOI: 10.5281/zenodo.18110161). Repository & Data:https://github.com/buk81/uniformity-asymmetry

Read Full Paperexternally

KI fragen

Bookmark

View Full Paper