Answers are grounded in this paper and cite their sources.
Retrieval-augmented generation (RAG) is increasingly used to ground large language model (LLM) tutors, but comparing tutoring architectures is expensive, because every design decision ordinarily requires a fresh study with students. We present a reproducible simulation harness that evaluates tutoring architectures against LLM-based synthetic learners, and we use it to run a full factorial comparison. Five learner profiles, parameterized from Conceptual Change Theory and the Force Concept Inventory literature, were crossed with four tutoring conditions (knowledge-graph-augmented RAG, BM25 RAG, a vanilla LLM without retrieval, and a non-LLM template baseline) across 30 deterministically seeded replications, giving 600 fully logged sessions on GPT-4o-mini, plus a 58-session cross-provider sensitivity run on Claude Sonnet 4. Two findings were robust to every check we applied. First, the learner-profile effect (ηp2=0.315) resolved into two clusters rather than a ranked gradient, and a Condition × Profile interaction (ηp2=0.097) showed that no architecture lifted the low-engagement, low-prior-knowledge profile. Second, every LLM-based condition exceeded the non-LLM floor under the simulator’s rule, under two independent LLM judges from different model families, and under a binomial generalized linear model with Holm-corrected inference. A third finding is measure-dependent and we report it as such: under the simulator’s keyword-triggered revision rule, the Misconception Revision Rate followed a monotonic retrieval gradient across the LLM-based conditions (M=0.40 for KG-RAG, 0.33 for RAG and 0.24 for the vanilla LLM; ηp2=0.099), but this gradient was not reproduced by either LLM judge (Cohen’s κ with the simulator =0.12 and 0.07) and was not reproduced in the Claude Sonnet 4 sensitivity run. A re-scoring ablation over the fixed dialogue logs shows that the gradient survives leave-one-keyword-out, IDF-weighted and independently authored vocabularies, and that the simulator’s numeric constants can be perturbed by ±50% without changing any ordering, so the gradient is a stable property of keyword-based measurement rather than of one keyword list, while its absence under the judges bounds how far it should be read as evidence about learners. All architectural findings are reported as simulator-internal results and as pre-registerable hypotheses for human-subject testing. The full pipeline, all session logs, both judges’ outputs and the analysis code are released.
No takes yet. Share an insight, caveat, or question.
Yumuşak et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: