What question did this study set out to answer?

The study aims to assess how well large language models (LLMs) answer medical multiple-choice questions across three languages.

March 26, 2026Open Access

Performance Evaluation of Large Language Models in Multilingual Medical Multiple-Choice Questions: Mixed Methods Study.

View Full Paper

Ask AI

Bookmark

The study aims to assess how well large language models (LLMs) answer medical multiple-choice questions across three languages.
Analyzed 114 medical multiple-choice questions from an online tool in German, French, and Italian.
Conducted quantitative performance analysis on LLMs from OpenAI, Meta AI, Anthropic, and DeepSeek.
Compared performance with different input languages and prompt languages.
Performed qualitative analysis on answer explanations for incorrect responses.
LLM accuracy varied between 64% and 87% based on model and language.
Significant performance differences noted, especially with German questions performing best (P<.01).
Prompts in English generally improved performance compared to language-matched prompts.
Qualitative analysis identified different reasoning errors in the leading models' explanations.

No takes yet. Share an insight, caveat, or question.

Mixed methods study evaluates LLMs' performance on medical questions in German, French, and Italian, suggesting implications for education.

Strasser et al. (2026) studied this question.

View Full Paper

Ask AI

Bookmark

Mixed methods study evaluates LLMs' performance on medical questions in German, French, and Italian, suggesting implications for education.