Answers are grounded in this paper and cite their sources.
LLMs are used worldwide, but whether they perform consistently across different languages is a question that has not been rigorously tested. This paper examines cross-lingual prompt sensitivity by running four models (Kimi K2 Instruct, Llama 3.3 70B, Qwen3 32B, and GPT-OSS 120B) on 48 semantically equivalent prompts across English, Spanish, Urdu, Japanese, and Arabic, covering six prompt categories and producing 960 total responses. The observed disparities were larger than expected. Japanese prompts received responses averaging 187.1 words versus 646.9 for English, a statistically significant 71% gap (Cohen’s d = 1.14, 95% CI [0.91, 1.37], p < 0.0001), and 26.6% of Japanese responses were classified as minimal compliance compared to 4–6% in the other languages. One-way ANOVA confirmed that response length varies significantly by language (F (4, 955) = 34.93, p < 0.0001, η² = 0.128, medium-to-large effect), and chi-square analysis confirmed the same for refusal patterns (χ² (12) = 100.54, p < 0.0001, Cramér’s V = 0.187). Model effects on response length were even larger (F (3, 956) = 128.57, p < 0.0001, η² = 0.287). Japanese responses showed the highest Type-Token Ratios (mean = 0.823, 95% CI [0.797, 0.849]), reflecting a measurement artifact from brevity rather than genuine vocabulary richness. All four models showed this pattern across all six prompt categories, making it difficult to attribute to any single architectural choice. Training data imbalances and tokenization inefficiencies are the most plausible explanations. For a technology marketed as globally accessible, these results raise questions worth taking seriously.
No takes yet. Share an insight, caveat, or question.
Nizami et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: