The Geometry of Language: Analyzing Input Complexity and Transformation Matrices in Large Language Models

Abstract In the evolving landscape of Large Language Models (LLMs) such as GPT-4, the effectiveness of methodologies like In-Context Learning (ICL) and the "Chain of Thought" (CoT) has been recognized for enhancing AI comprehension and interaction. These approaches align with prompt engineering principles—emphasizing clarity, role-prompting, and structured prompt design to reduce grammatical complexity and use familiar language, thus improving LLMs' interpretative accuracy and response consistency. Our study introduces a mathematical framework to conceptualize LLM parameters as transformation matrices, interpreting textual complexities into high-dimensional vector spaces. This framework helps understand how prompt structure affects LLM performance by categorizing prompts from simple to complex and examining their impact on the intrinsic dimensionality of LLMs.Findings show a direct correlation between prompt complexity and LLM intrinsic dimensionality. Simple sentences lead to lower dimensionality (11.78, 9.99, and 9.60 for models of 7B, 13B, and 33B respectively), while grammatical complexity increases it (12.08, 14.08, and 12.70), and linguistic complexity yields lower or comparable values. Oue research bridges the knowledge gap on the mechanics behind ICL and CoT's efficacy, offering strategies to optimize LLM capabilities. It provides a fresh perspective on LLM behaviors and underscore the critical role of prompt design in maximizing the potential of these powerful computational tools across various applications.