Spring Embeddings: A Physics-Based Diagnostic Tool for Language Model Semantic Representations

We introduce Spring Embeddings, a physics-based method for analyzing and visualizing the internal semantic representations of language models. By modeling tokens as vertices in a fully-connected graph with spring forces derived from embedding similarities, we reveal semantic structures that are invisible to standard dimensionality reduction techniques. Applying Hooke's law with dual spring mechanics (attraction and repulsion), we project the model's knowledge into interpretable spatial configurations. We extend this by projecting embeddings through extracted Q, K, and V weight matrices from transformer attention layers, revealing three distinct semantic lenses: Query space (what a word seeks), Key space (what a word offers), and Value space (what information a word carries). Our analysis across three embedding models (nomic-embed-text, all-minilm, mxbai-embed-large) yields five novel findings: (1) attention asymmetry between Q and K spaces, (2) universal gravity centers that attract semantically diverse words, (3) V-space superiority for categorical clustering, (4) cross-lingual Q-alignment with +14.9% EN-RU similarity boost, and (5) spelling bias detection revealing 5.3x stronger orthographic clustering in Russian versus English embeddings. Code available at https://github.com/helgard-orlm/spring-embeddings