Answers are grounded in this paper and cite their sources.
Benchmark evaluates language model performance on German business tasks, indicating areas for improvement.
Most public evaluations of large language models (LLMs) test English-language capability; German is usually covered only through translated test sets that do not probe German domain knowledge, registers, or administrative language. KI-Benchmark Deutsch is a recurring benchmark that evaluates leading LLMs on realistic German business tasks across six categories: formal business correspondence, official and bureaucratic German (Amtsdeutsch), verifiable law and tax questions, faithful summarisation, source-grounded answering, and language quality. Subjective categories are scored 0–100 by a cross-vendor panel of neutral judge models under a leave-one-family-out, name-blind protocol with manual spot checks; reference categories are scored against ground truth, with hallucinated statutes or deadlines counted as wrong. The test set is held out; only sample prompts are public. This note documents the benchmark design, scoring methodology, contamination controls, and limitations, and summarises the run of 2026-06-22 (7 models, 24 tasks per model), in which the weakest category across every model family was official and bureaucratic German (category mean 83.4/100). Results are published as an open dataset (CC BY 4.0, DOI 10.5281/zenodo.21452970); the live leaderboard is freely available at https://i6eal.de/ki-benchmark-deutsch/.
No takes yet. Share an insight, caveat, or question.
Ideal Syka (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: