What does this research mean for the field?

Frontier LLMs impose a severe structural penalty on lower-resource Asian languages, resulting in up to an 11.7x cost increase and a near-total collapse in context recall for non-Latin scripts at just 4,000 tokens. Novelty: ClaimNovelty.INCREMENTAL. Consensus alignment: ConsensusAlignment.NEUTRAL.

What question did this study set out to answer?

This work aims to quantify the structural penalties imposed on speakers of lower-resource Asian languages by tokenization in large language models.

July 3, 2026Open Access

Ver artículo completo

The Asian Language Tax: Quantifying the Cost, Context, and Recall Penalty of Tokenizing Lower-Resource Asian Languages in Frontier LLMs

APAntoine Pedretti

Puntos clave

This work aims to quantify the structural penalties imposed on speakers of lower-resource Asian languages by tokenization in large language models.
Analyzed 16 Asian languages across seven families using parallel corpora to isolate language effects.
Reported same-content cost ratios using FLORES-200 dev with bootstrap confidence intervals (n=997).
Introduced a bytes-per-token metric for fair cross-script comparisons.
Median cost ratio of 8.9x on cl100k_base for 11 non-Latin scripts, peaking at 11.7x for Burmese.
Recall for non-Latin scripts dropped to 0-2 out of 10 at 4k tokens for 4 out of 5 models tested.
Notable finding: gemini-2.5 achieved 84.5% pooled recall, indicating variability in model training choices rather than inherent limits.

Resumen

Commercial large language models bill, throttle, and budget context per token. Yet tokenizers assign more subword tokens to the same meaning in some languages than in others, so speakers of high-fertility languages pay a structural penalty before a model is ever invoked. This work measures that penalty across 16 Asian languages spanning seven language families and twelve scripts (Latin, Devanagari, Bengali, Sinhala, Tamil, Telugu, Kannada, Malayalam, Myanmar, Khmer, Lao, Thai), using parallel corpora so that the language effect is isolated from content. Contributions: - Same-content cost ratio across 16 Asian languages and 10 production tokenizers on FLORES-200 dev (n=997), with bootstrap 95% confidence intervals. Median 8. 9x on cl100kbase for the 11 non-Latin scripts, up to 11. 7x for Burmese. - Bytes-per-token (BPT) metric introduced for cross-script-fair comparison. - A 4, 000-cell needle-in-haystack recall benchmark with script-native markers, across 5 frontier OpenRouter models x 16 languages x 5 fill levels up to 131k tokens. - Headline finding: recall on non-Latin scripts collapses to 0-2/10 already at 4k tokens for 4 of 5 tested models — well inside every model's context window. gemini-2. 5 lash breaks the pattern with 84. 5% pooled recall on the same grid, indicating a vendor-level training/serving choice rather than an intrinsic limit. - Wall-clock latency penalty benchmark (800 measured trials + 240 warmup calls). Pooled Pearson r between costᵣatio and latencyᵣatio = 0. 314 — notably weaker than prior African-language findings — with per-model range -0. 19 (qwen) to +0. 74 (llama-3. 1-8b). Release artefacts: - Python package: https: //github. com/Helmo21/asia-fertility (MIT) - HuggingFace dataset: https: //huggingface. co/datasets/Helmo21/asia-fertility (CC-BY-SA 4. 0) - Manifest with SHA256 of config, pinned price + FX snapshots, tokenizer versions per run. This record bundles the preprint (PDF + LaTeX source), the three benchmark CSVs (leaderboard, NIAH v0. 3, latency), and the study manifest so any figure in the paper can be reconstructed byte-for-byte.

Preguntar a la IA

Me gusta

Guardar

Ver artículo completo

Preguntar a la IA

Me gusta

Guardar

Ver artículo completo

The Asian Language Tax: Quantifying the Cost, Context, and Recall Penalty of Tokenizing Lower-Resource Asian Languages in Frontier LLMs

Puntos clave

Resumen

Cite This Study

Also Consider

Also Consider