The Architecture of Linguistic Discretisation: A Comparative Analysis of Tokenisation Strategies in Large Language Models

This article provides a comprehensive analysis of linguistic discretisation, the foundational process of converting continuous language into numerical tokens for Large Language Models (LLMs). The choice of tokenisation strategy is not a mere preprocessing step but a critical architectural decision that fundamentally influences a model's computational efficiency, morphological understanding, and inherent biases. The analysis traces the evolution from early word-level methods, which suffered from unmanageable vocabulary sizes and out-of-vocabulary errors, to character-level methods, which were computationally prohibitive due to long sequence lengths. This historical context establishes the rationale for the modern dominance of subword algorithms such as Byte-Pair Encoding (BPE), which seek to balance vocabulary granularity and sequence length. An empirical experiment demonstrates that even advanced BPE implementations can be less efficient at sequence compression than simpler heuristics, revealing hidden flaws in how algorithms handle whitespace and common word fragments. These flaws manifest as significant cognitive artefacts in production models, such as failures in character-level reasoning, fragmented arithmetic processing, and a "multilingual token tax" that disadvantages non-English languages. Counterintuitively, the analysis shows that maximising compression does not necessarily yield superior model performance; rather, aligning tokens with natural linguistic structures is more critical. The report concludes by exploring the architectural frontier beyond traditional tokenisation, examining innovative approaches like "superword" models that bridge whitespace to capture multi-word expressions and token-free Byte-Level Transformers that process raw data streams, promising a future of more efficient, equitable, and architecturally robust language processing.