AHSPE: Adaptive Hierarchical Semantic Patch Encoding for Efficient Tokenization of Language Models

Byte Pair Encoding still runs tokenization for most large language models, and it has a well-known blind spot: it merges characters by frequency, not by meaning. So “machine learning model” gets split however the statistics fell during training, regardless of whether that split makes any linguistic sense. I built AHSPE—Adaptive Hierarchical Semantic Patch Encoding—to fix this. The idea is to group text into variable-length patches that actually cohere semantically, organized across three levels: characters, subwords, and multi-word phrases. A learned scoring function picks which patch to use at each position by weighing corpus frequency, how well the candidate fits the surrounding context, byte-level entropy, and howmany tokens the choice saves. Crucially, the tokenizer is not trained in isolation—it is optimized jointly with the languagemodel using a rate-distortion objective, so compression and prediction quality are traded off explicitly. On a mixed text-and-code corpus, AHSPE gets down to 0.361 tokens per character 2.77× fewer than character-level encoding, with a vocabularyan order of magnitude smaller than standard BPE. Fewer tokens means shorter sequences, and shorter sequences mean lessquadratic attention cost at inference time.