"It Depends," But Now We Can Measure Why: Benchmarking Specialised and General-Purpose AI on Humanities Tasks

Poster presented at the Transkribus User Conference 2026, "Not all AI is created equal. Unlocking the written past with specialised and general-purpose AI" (University of Passau, 21-23 September 2026). The poster asks whether a specialised handwriting model transcribes a project's own manuscript pages more accurately than a general-purpose vision-language model, and settles the question by measurement rather than by tool category. Twelve selected Transkribus models and 104 general-purpose models were run on the same twelve pages of mid-15th-century German (Basel, Universitätsbibliothek, H V 15) and scored against the same expert-prepared reference with one character error rate function. The best general-purpose model reached 0.061 mean page CER at USD 0.83 for the twelve pages; the best selected Transkribus model, Text Titan II, reached 0.130 at six Transkribus credits. Fourteen of the 104 general-purpose models improved on it, but the category median was 0.266, higher than ten of the twelve selected Transkribus models, and three general-purpose models scored 1.000. Tool category alone does not select an accurate model. The remaining sections document what makes the comparison comparable (one reference and one scorer for both kinds of tool, a prompt asymmetry bounded at under two CER points, equal page weights, missing or unparsable output scored 1.0 and left visible, each model represented by its best of 147 recorded runs), a sensitivity analysis removing from both sides the one page on which the shared layout analysis fails (0.053 against 0.094, eleven models of 104), a three-step shortlist for testing a new collection, and the open RISE Humanities Data Benchmark the protocol comes from: 12 humanities datasets, 123 models, 12 providers, covering transcription, information extraction, document understanding and data correction. Figures as of 4 September 2026. A0 poster, English. Code: https://github.com/rise-unibas/humanities_data_benchmark/Dashboard: https://rise-services.rise.unibas.ch/benchmarks/