Byte Patches versus BPE at Matched Data: A Matched-Twin Comparison at 350M Parameters

We compare a tokenizer-free byte-patch language model with a BPE language model at 350M parameters, trained on the same 10B bytes of text, code and math, with similar compute (about 2% FLOPs difference). The byte-patch model uses a 4-byte patch encoder, a 26-layer global transformer and a small local byte decoder with a 64-byte window. The BPE twin uses an 8,000-token byte-level vocabulary and the same 26-layer backbone. Under a matched-span evaluation with paired bootstrap confidence intervals, the BPE twin reaches lower bits-per-byte overall (0.9155 vs 0.9634, about 5%), while the byte-patch model is significantly better on a character-level benchmark. We report the limits openly: one model size, one seed, one fixed vocabulary, and a known difference in embedding initialization between the two models, for which a control run is in progress. Code, weights and validation data are public.