Technical report, version 1 (preliminary, not peer reviewed). We compare the marks of a language model (GPT-6 Astra at its highest reasoning setting) with the marks agreed by human juries on 2,230 proof attempts from 21 MathJack olympiad contests. On contests where no AI mark was shown to the jury, the model matched the jury mark exactly on 49% of attempts and was within one point on 81%; it was closest to the jury on zeros and full marks, and matched exactly 27% of partial-credit marks (72% within one point). We also report a prompt-strictness experiment, an error analysis, and a controlled probe with planted slips and gaps, and outline next steps: independent human re-grading, replays across several models, evaluation of a routing policy, and human-verified synthetic slip and gap proofs.
No takes yet. Share an insight, caveat, or question.
Hashir Naimat (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: