Analysing hospital responses to patient-reported safety incidents online using large language models

Patient experiences reported online can reveal safety incidents in healthcare organisations, but they can only translate into quality improvement if organisations engage with them. Accordingly, healthcare organisations’ responses to online feedback offer a new data source on corrective culture, namely, the norms and practices for identifying, interpreting, and acting on safety incidents. This study explored the feasibility of using a large language model to reliably code these responses for two candidate indicators of corrective culture: engagement and organisational learning. We manually coded 600 feedback-response pairs for engagement (κ = 0.90) and organisational learning (Gwet’s AC1 = 0.84), and used this gold standard to develop and evaluate a GPT-5-mini classifier. The classifier coded engagement with high agreement (F1 = 0.86), but was less reliable for organisational learning (F1 = 0.70). To assess construct validity, we analysed human disagreements and classifier misclassifications. Organisational learning was harder to code because it is a reported behaviour, whereas engagement is an observed behaviour. We then used the classifier to code 157,051 responses (2011–2023) across 844 healthcare organisations. Our results indicate that large language models can reliably code engagement in these responses, while their ability to code organisational learning remains inconclusive. Organisational culture in healthcare is usually assessed with staff surveys, which rely on self-report. Large language models could complement surveys by measuring observed behaviour in naturally occurring data, although whether engagement reflects corrective culture requires external validation.