This is the authors' abstract. We don't add key points for this paper.
While frontier multilingual Large Language Models (LLMs) impose a severe tokenization tax on morphologically rich, low-resource Indic scripts such as Odia (ory_Orya)—fragmenting words into 5-6x more subword tokens than English and inflating inference latency—compact, script-native models remain underexplored. We present OdiaSLM-60M (62.11M parameters) and OdiaSLM-100M (100.69M parameters), two sovereign decoder-only Small Language Models pre-trained from scratch for Odia on up to 1.57 billion tokens drawn from a strictly deduplicated and 13-gram benchmark-decontaminated corpus of 9.62 million documents (1.247 billion words). First, we design a 16,384-vocabulary canonical NFC-normalized SentencePiece BPE tokenizer that preserves Odia juktakshara conjuncts, achieving a fertility of 1.4809 tokens/word (11.57 UTF-8 bytes/token), 0 <unk> tokens, and a 3.6x compression advantage over general multilingual tokenizers. Second, utilizing Grouped-Query Attention (GQA), SwiGLU, Half-Split RoPE, FP32-variance RMSNorm, auxiliary Z-loss stabilization, a Warmup-Stable-Decay (WSD) schedule, and two-stage context extension (512 to 1024), we demonstrate clean neural scaling from 62.11M (validation perplexity 19.92, Bits-Per-Byte 0.3965) to 100.69M parameters (validation perplexity 17.14, Bits-Per-Byte 0.3767, Top-1 token accuracy 44.61%). Third, we conduct a comprehensive zero-shot evaluation across 27 tasks in the Odia Evaluation Benchmark, uncovering both cross-lingual untranslated choice artifacts in 9 tasks and severe surface-form competition under raw log-likelihood. Applying Pointwise Mutual Information (PMI) calibration unlocks a macro-average accuracy of 60.83% across the 18 pure-Odia benchmarks (+34.44% lift over random chance). Both models are released in native LlamaForCausalLM format.
Answers are grounded in this paper and cite their sources.
No takes yet. Share an insight, caveat, or question.
Arun Misra (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: