How much semantic information is available in large language model tokens?

Large language models segment many words into multiple tokens. For example, GPT-4 segments "dogcatcher" into dog+catch+er. Companies that make those models claim that meaningful subword tokens are essential, yet tokens often appear meaningless or misleading. For example, GPT-4 segments "anteater" into ante+ater, and those tokens don’t align with morphemes (i.e., ant+eat+er). To investigate whether tokens bear meaning, we segmented tens of thousands of words from each of 41 languages according to three generations of GPT tokenizers (GPT-2, GPT-4, and GPT-4o). We found that words which share tokens are more semantically similar than expected by chance or expected from length alone, that tokens capture morphological information even when they don’t look like morphemes, and that tokens capture more information than is explained by morphology. These results suggest that comparing tokens to morphemes overlooks the wider variety of semantic information available in word form and that standard tokenization methods successfully capture much of that information. However, tokens convey less semantic information in lower resource languages and in languages that don't use the Latin alphabet, so standard tokenization methods might entrench advantages for speakers of English and other high-resource languages.