Shared Latent Geometry Across Model Scales: Procrustes Spectral Alignment, Scale Splitting Index, and Causal Activation Patching in Small and Frontier LLMs

Overview This working paper and experimental protocol by Jeremiah Swann (Lead Researcher) investigates the hypothesis that large language models (LLMs) trained via stochastic gradient descent converge toward a universal semantic manifold—a shared latent space—because human language itself evolved as a lossy compression algorithm of physical reality. Building on the Platonic Representation Hypothesis (Huh et al., 2024), Information Bottleneck analyses of semantic compression (Shani et al., 2025), Relative Representations (Moschella et al., 2023), and unsupervised cycle-consistent embedding translation via vec2vec (Jha et al., 2025), this work formalizes how smaller-parameter models (70M–2B parameters) embed as capacity-constrained projections of frontier reference manifolds (4B–70B+ parameters). Core Theoretical & Methodological Contributions World-Compression & PMI Kernel Convergence: Formalizes how natural language lexicalization under an Information Bottleneck objective (Tishby et al., 2000) preserves the Pointwise Mutual Information (KPMI) topology of physical events, driving disparate neural architectures toward an isometric inner-product kernel in intermediate residual-stream layers. Scale Splitting Index (SSI) & Superposition Disentanglement: Extends cross-scale Sparse Autoencoder (SAE) universality benchmarks across Pythia (Biderman et al., 2023) and Gemma-2 (Riviere et al., 2024) established by Lan et al. (2025) and Son et al. (2025), defining the cumulative Scale Splitting Index (SSIcum) to quantify surjective (ManyF-to-1S) feature splitting across parameter scales. Orthogonal Procrustes Spectral Trajectory & Random Control Baseline (ΔProcrustes): Formulates closed-form Orthogonal Procrustes alignment (Procrustes Bounds, 2025) and nuclear-norm singular value decay trajectories across normalized network depth, paired against a randomly initialized Pythia-70M negative control group (Mₙᵤₗₗ) to prove that structural isomorphism is a learned consequence of optimization over the data distribution rather than architectural initialization. Cross-Scale Causal Activation Patching & Compression Breakdown Boundary (^*): Grounds causal functional isomorphism in localized mid-layer MLP associative memory editing (ROME, Meng et al., 2022a; MEMIT, Meng et al., 2022b), projecting frontier residual updates into smaller experimental cohorts (Pythia, Gemma-2, and Qwen2, Yang et al., 2024) to isolate the exact layerwise threshold (^*) where post-patching KL reconstruction loss (Lpatch) spikes and geometric compression breaks down.