Answers are grounded in this paper and cite their sources.
Pilot benchmark evaluates large language models on Kazakh language tasks, highlighting differences in performance.
The performance of large language models (LLMs) on Kazakh — a low-resource language — is poorly characterised, and existing evaluations typically reduce to a single aggregate score. We hypothesise that "language understanding" is not one-dimensional, and, within a pilot benchmark, provide evidence supporting this hypothesis. Track A (fact-checking) probes reasoning over a provided text: given a source and a claim, the model returns a verdict of SUPPORTED / REFUTED / NOT_ENOUGH_INFO; 60 hand-built claims span three genres (legal, a news report, and an informal messenger dialogue mixing Kazakh and Russian). Track B (idioms) probes closed-book knowledge of Kazakh phraseology over 30 idioms in two phases (without context and, on failure, with a usage example). Five models from five labs (Claude, Gemini 2.5 Flash, Qwen3.7-Plus, DeepSeek-V3, Kimi K2.6) are evaluated uniformly; in Track A every counted run passed an automatic grounding check. The main observation is a dissociation of abilities for the studied models. On Track A all five score 92–97% and are statistically indistinguishable (overlapping confidence intervals; pairwise McNemar p ≥ 0.375). On Track B results range from 28% to 82%, and the ranking reorders: the model ranked second on fact-checking (Kimi, 95%) is last on idioms (28%). A single aggregate score would, in all likelihood, not have surfaced this difference. We also note a characteristic failure of weaker models: confidently substituting a real Russian idiom with the wrong meaning. All materials (prompts, raw model outputs, grades, data, code) are open. The observations are bounded by the sample size and model set of this pilot and are not claimed to generalise to LLMs at large.
No takes yet. Share an insight, caveat, or question.
Timur Seidalin (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: