Answers are grounded in this paper and cite their sources.
Large language models (LLMs) are increasingly used to answer questions about predictive models, but a fluent answer about a model is not necessarily true of it. We propose a framework in which every answer is checked against a reference that independent software computes from the fitted model. The main finding is simple: the LLM was much more accurate when software had done the calculations than when it had to work them out from the model itself. In a confirmatory study with Gemini 2.5 Flash, the LLM correctly answered 16 of 36 questions whose answer follows exactly from the full model description, such as a prediction or a decision path, but 33 of the same 36 once software had computed the values from which the answer follows. A replication with 50 classifiers, five datasets and three LLM configurations, including a reasoning setting and an open-weight model, reproduced this pattern. Without model information, the LLMs declined to answer or guessed at chance level, and verified answers were reported faithfully once their origin was stated. Model-specific calculations should therefore be left to verified software, while the LLM reports and explains the checked results.
No takes yet. Share an insight, caveat, or question.
Ficsor et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: