Answers are grounded in this paper and cite their sources.
Explainability for large language models (LLMs) spans circuit analysis, feature discovery, attribution, probing and editing, rationale faithfulness, source attribution, benchmarks and human-facing explanation. This systematic mapping study charts studies that explain an LLM or an LLM-centred pipeline, excluding work that utilises an LLM to explain another system. Seven sources were searched for reports from 2019 to August 2026 that explain at least one decoder-only or encoder–decoder model of at least 1B parameters. Of 7,825 records (5,730 after deduplication), 1,252 studies were included and coded by method family, explanandum, required access, evaluation construct and a four-field evidence profile. Screening, eligibility and coding were performed by a disclosed AI reviewer whose every decision carries a verbatim quote, and were validated against a human reviewer: title/abstract agreement reached κ = 0.84 after two failed calibration rounds, full-text agreement κ = 0.72, and conservative screening recall was 0.78. Methods requiring weights or activations account for 69.8% of the studies and methods compatible with hosted text or logit access for 23.3%; faithfulness or causal-validity evidence is coded in 70.4% (an approximate AI-coded estimate), human utility in 3.4%, deployed settings in 7 studies and specific regulatory obligations in 10. The map indicates that most explanation methods require access that hosted LLMs do not expose, and that human-centred and deployment evidence remains scarce. The protocol, its amendment and every decision are released.
No takes yet. Share an insight, caveat, or question.
Makridis et al. (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: