Development and refinement of a FMECA framework for identifying patient safety risks in LLM-generated clinical summaries: a methodological pilot study

Background Claims that large language models (LLMs) improve clinical care are outpacing the evidence needed to support them, and there is a lack of structured methods to produce the safety-specific evidence required before their outputs are trusted in practice. Failure Mode, Effects, and Criticality Analysis (FMECA) is medicine’s established route to this kind of proportional, pre-deployment safety evidence, but, to our knowledge, it has never been adapted to generative AI. This methodological pilot study aims to develop and refine the first FMECA framework for identifying and characterizing potential patient safety risks in LLM-generated clinical summaries. Methods An interdisciplinary panel ( n = 8) developed, through a two-round iterative expert consensus, a taxonomy of failure modes and adapted the standard FMECA dimensions (occurrence, severity, detectability) into 5-point ordinal scales. To refine the FMECA framework, 36 clinical summaries were generated by an open-weight LLM (GPT-OSS 120B) using data from Geneva University Hospitals: Three clinical reviewers annotated the summaries across two rounds. As exploratory secondary analyses, preliminary internal inter-rater reliability was assessed at the failure-mode, severity, and detectability levels, and internal usability was evaluated with an adapted System Usability Scale (SUS). Results From 78 candidate failure modes, the final taxonomy comprised 14 failure modes across six categories, with operational severity, detectability and occurrence scales. In the exploratory analyses, inter-rater agreement improved between rounds, reaching moderate-to-substantial agreement for failure-mode identification and good agreement for severity and detectability scoring; internal usability was rated good (SUS 79.2/100). Conclusions We propose a FMECA framework for the systematic identification and characterization of failure modes in LLM-generated clinical summaries. Occurrence and risk priority were deliberately not estimated, as the sample was designed for framework refinement: this taxonomy will highly benefit from large, empirical validations.