Answers are grounded in this paper and cite their sources.
Background : LLMs increasingly draft personalized academic outreach, which depends on describing a real professor's research. Unsupported claims about a recipient's topics, methods, or findings misrepresent their scholarly work. Methods : In an offline pilot, a 2 × 2 design varied whether the writer received a publication evidence packet during drafting and whether a verifier revised the draft afterwards (conditions A–D). We generated 96 emails for 12 professors across three fields, decomposed them into 360 atomic research claims, and labelled each against the professor's evidence packet with an automatic (LLM) evaluator. All primary estimates are pilot estimates from automatic labels, not human-verified measurements. Reliability was assessed against a blinded 36-claim human-labeled subset (10% of claims) and a second automatic evaluator from a third model family. Results : Under the primary evaluator, the closed-book baseline (A) had a Severe Error Rate (SER) of 0.344 (21/61 claims; exact 95% CI 0.227–0.477). Grounding the writer (B) produced no severe errors in 90 judged claims (0.000, CI 0.000–0.040) and raised Supported Personalization Density (SPD) from 0.65 to 1.71 supported claims per 100 words. Verify-only (C) left 1 severe error in 50 claims (0.020; SPD 1.04); grounded-plus-verified (D) produced none (0/87) at the highest specificity (1.76). Interpretation and limits: Primary-evaluator agreement with the human annotator was only fair (Cohen’s κ = 0.33, CI 0.16–0.49). The evaluator was too lenient in 14 of 16 disagreements, suggesting that the automatic severe-error rates may underestimate error prevalence. However, the small human-labelled subset does not establish the magnitude or direction of bias across the complete dataset. The zeros are evaluator-specific: a second evaluator relabelling the same claims found 10 severe errors in B and 4 in D, closed-book 0.455 - a large reduction under grounding, not elimination. And 17 of the 21 closed-book severe errors came from the four psychology professors; excluding that field, closed-book SER falls to 0.095. Both automatic evaluators reported lower pooled severe-error rates for grounded conditions than for ungrounded conditions. Grounding during drafting also showed a lower observed pooled SER than post-generation verification alone. Conclusion : Verified publication evidence sharply reduced severe misrepresentation - to zero under the primary evaluator, roughly four-fold under the second - while making emails more specific, and outperformed correcting drafts afterwards. Grounding, not correction alone, is the more promising safeguard; establishing the effect size needs human labelling at scale and a validated judge.
No takes yet. Share an insight, caveat, or question.
Mammadov et al. (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: