Evaluating a large language model–driven safety culture interview chatbot and transcript analysis in a simulated experiment

Safety-critical industries are placing increasing attention on the cultural aspects of safety. Conducting safety culture assessments has become commonplace for supporting the continuous improvement of safety. Comprehensive safety culture assessments are highly resource-demanding and complex projects that involve qualitative data collection and analysis. In this article, we examine the applicability of tools driven by large language models (LLMs) to conduct and analyse safety culture interviews. Firstly, we developed a custom, LLM-driven safety culture interview chatbot. The chatbot was used to create a simulated safety culture interview dataset where it interviewed a human safety culture expert roleplaying ten interviewees. Secondly, the interview transcripts were analysed in parallel by the LLM and by another human expert, and the results of the analyses were compared. We found that the chatbot was successful in collecting useful information within the safety culture interview context. It appears to complement human-led safety culture interviews and safety climate questionnaires by offering a scalable, yet qualitative and interactive, data collection solution. We also found that the LLM was useful for identifying safety culture themes from the interview transcripts, offering an increase in efficiency and potentially offering an alternative perspective on the data. We conclude that while LLM-based tools can be valuable for safety culture assessments, they should complement, rather than replace, human involvement and the established methods. Based on our observations, we propose several future research needs.