Small Dragon Hatchling Models: Alphabet, Memory, and Fine-tuning

I trained a set of Dragon Hatchling models of sizes 25M, 50M, and 100M parameters on two alphabets, bytes and Byte Pair Encoding (BPE) tokens, using the same text at 25M and 50M and the same amount and mix of text at 100M. BPE outperforms bytes by 4 to 8% in bits per character on the same held-out text. The two alphabets score within 0.02 bpc of each other when given roughly the same amount of context, which suggests that part of the gap comes from each BPE token position seeing four times more characters. Compared to a rotary transformer of matched size trained on the same amount and mix of text, BDH scores about 5% better in bits per character on bytes, at about six times the compute per position. On BPE, the two are within 2.4% of each other at every size, so the byte-level advantage mostly disappears. I built streaming inference and found that feeding each position once into the synaptic state ρ matches recomputing all the previous context, and with a sliding context window the model predicts as well as a fresh model given the last 480 positions, through 4,096 positions of text, which is past the 512-position block it was trained on. I tested recall by planting a fact in each model’s context. In the BDH base models, the planted sentence makes the word hundreds to tens of thousands of times more likely at distances up to 128 characters. The BDH byte base models mostly fail to recall the correct word while the transformers do. However, chat tuning the models teaches them to do so, raising byte recall from about 17% to 93% at 128 characters. For the byte model only the v3 data produced this, whereas both the v2 and v3 tuning sets worked for BPE. Compared to v2, v3 added list items and changed the story questions at the same time, so I did not determine which change produced this. At 100M, the distance the planted fact can be recalled from is several times larger with BPE than bytes after tuning. LLM judges comparing single replies cannot separate the v2 or v3 models or the two alphabets, but a multi-turn exam can. When asked questions about a story it has just written, the tuned BPE model gives answers supported by that story 36% of the time, and the tuned byte model 16% of the time. By varying the number of layer passes at inference, I found that BDH models need to run inference at the same number of layers they were trained on, otherwise their output quality drops rapidly. Looking at the neuron density of the 25M, 50M, and 100M models, the larger models fire a smaller proportion of their neurons. Compute and budget were limited, so most measurements come from a single training run per model. The code, result files, and transcripts are at github.com/d3vmeh/small-dragon-hatchling, and the six base models and the two v3 chat models are at huggingface.co/collections/d3vmeh/small-dragon-hatchling-models.