Abstract Objectives Timely access to pathology reports has increased the need for clear patient-facing explanations. We evaluated whether large language model (LLM)–generated responses to pathology report questions from patients are comparable in quality to explanations written by pathologists and assessed how LLM configuration influences performance. Methods Sixty-five anonymized real-world patient questions from an online pathology education platform were answered using 5 LLM configurations varying by model architecture, prompting strategy, and retrieval-augmented generation. Responses were evaluated using a structured rubric that assessed accuracy, relevance, clarity, empathy, and safety; they were compared using pairwise arena testing with Bradley-Terry modeling to rank performance. Results Across rubric domains, LLM responses demonstrated performance comparable to pathologist explanations, with 1 configuration meeting noninferiority criteria. Pairwise arena comparisons indicated that the configuration parameters strongly influenced performance, with both model size and retrieval augmentation associated with improved response preference. The highest-performing configuration combined a larger model with retrieval from a curated pathology knowledge base and was strongly preferred over pathologist-written responses. Conclusions Carefully configured LLM systems can generate patient-facing explanations of pathology reports comparable in quality to pathologist-written explanations. Prompting strategy, model size, and retrieval integration were associated with performance differences, underscoring the importance of system configuration in developing LLM-based tools to expand access to understandable pathology information.
Jeong et al. (Mon,) studied this question.