Language Asymmetry in Multilingual Clinical AI: A Benchmark Study of LLM Performance in ICU Decision Support Across English, Slovak, and Ukrainian

Background. Most clinical LLM testing and benchmarking is carried out in English, on cloud infrastructure, and under conditions that bear little resemblance to a real hospital. How these models perform in other languages, and under substantially constrained compute — particularly in intensive care units, where the cost of error is always high — remains largely unknown. Objective. The question we set out to answer looks simple: does language matter when AI is used to solve clinical problems, and if so, how does it affect the solutions? To find out, we tested a locally deployed RAG pipeline in English, Slovak, and Ukrainian on identical synthetic intensive care cases. Methods. After extended experimentation we built a three-agent clinical decision support system (Clinician → Expert → Judge) running on consumer hardware (2× NVIDIA RTX 3060 12 GB, 192 GB RAM), named GALATEA III. Using this system we ran a benchmark of 6,600 evaluations of synthetic intensive care cases (100 cases per domain, three languages) across four optimisation phases, the last of which completed the migration from Gemma 3 12B to Gemma 4 26B. The synthetic clinical cases were generated with GPT-4o and reviewed by the author before use. Full case structure and evaluation metric definitions are given in the Supplementary Material. Five domains were run in all phases (1,500 evaluations each), disaster medicine from phase 2b onward (1,800 each). Results. The Gemma 3 baseline showed pronounced language asymmetry: the APPROVE rate (means across the five common domains) was EN 40.0% / UK 48.4% / SK 18.6%. These figures incorporate a rule that uplifted a REVISE verdict to APPROVE when the score was ≥ 4.0 and plan safety was confirmed (Section 2.4). Without that rule — that is, on the Judge’s raw verdicts — the rates were 1.4% / 1.4% / 1.2%: at baseline the Judge approved almost nothing in any language. Iterative optimisation narrowed but did not close the gap. Migration to Gemma 4 almost entirely removed the asymmetry across all domains (EN 99.7% / SK 97.8% / UK 98.5%; for Gemma 4 both definitions coincide). Qualitative analysis revealed distinct reasoning styles by language: analytical-argumentative in English, procedural-directive in Slovak, and narrative-empathic in Ukrainian. The conflict detection metric, which does not saturate on Gemma 4, revealed a gap invisible to the APPROVE rate: 100.0% in English and 97.5% in Ukrainian against 5.4% in Slovak. Inspection of the response texts established that this is a structured-reporting failure rather than a detection failure: in 30 of 30 cases reviewed manually by the author, the Slovak Judge described the conflict in prose while recording the opposite tag. Conclusions. Clinical LLM performance is tied to training data volume rather than to language relatedness: corpus size reliably separates English from both lower-resource languages, whereas the relative position of Ukrainian and Slovak with respect to each other is unstable and this study does not resolve it. With sufficient model capacity the gap between languages narrows by more than an order of magnitude — from 47.8 to 2.2 percentage points. These results call for targeted development of medical corpora for lower-resource European languages. Separately, we show that multilingual benchmarks relying on a model’s structured self-reports may be measuring formatting discipline rather than clinical competence.