Generative artificial intelligence for inpatient documentation summarization: mixed-methods quality assessment and early real-world experience

Abstract Objective We evaluated the quality and adoption of a large language model (LLM)-based summarization tool for ongoing hospital care. Materials and methods An AI summarization tool was created to provide a “Patient Story”, specialty-specific “Recent Notes”, and “Recent Events” over the previous 24 or 72 hours. We conducted a pragmatic mixed-methods quality assessment at three tertiary-care academic hospitals utilizing (1) Provider Documentation Summarization Quality Instrument (PDSQI-9), (2) utilization analytics, and (3) qualitative end user feedback. The PDSQI-9 included whether summaries were accurate, cited, comprehensible, organized, succinct, non-stigmatizing, synthesized, thorough, and useful. 512 users were given access, from whom 52 participants submitted 205 surveys (10.2% response rate). Results 52 respondents submitted an average of 4.3 surveys (range 1–8). Users rated the tool favorably across all PDSQI-9 domains, with a combined average score of 4.68 (range 4.56–4.78, S.D. 0.67) across the eight domains scored on a 5-point modified Likert scale. Utilization metrics demonstrated strong uptake with frequent views. Positive qualitative feedback revealed cognitive offloading for complex patients and effective summarization of medical problems. Critical feedback showed a need to cross-reference narrative notes to current-state data and lack of detailed specialty-specific summarization. Discussion Generative artificial intelligence has emerged as a potentially transformative technology for generating succinct, verifiable summaries of ongoing care. In this pragmatic implementation, end-users indicated high perceived quality across PDSQI-9 domains. Conclusions An LLM-based summarization tool for ongoing hospitalization care was rated of high quality by diverse clinicians in real-world settings, demonstrated a favorable safety profile, and showed sustained utilization.