Answers are grounded in this paper and cite their sources.
Discussion highlights challenges of responsible LLM tool usage in safety engineering, suggesting critical implications for practice.
While Large Language Models (LLMs) have demonstrated significant impact on numerous engineering tasks, they introduce critical risks in safety-critical domains that cannot be adequately managed through bans, restrictions, or existing standards alone. This paper analyzes the challenges and limitations of applying LLMs in safety engineering activities, emphasizing the crucial distinction between standalone LLMs and comprehensive AI systems. In Section 2, we outline the key risks introduced by LLMs, including issues related to data quality, model architecture, operational constraints, output bias, non-determinism, failures in long-context processing, and more. To address these risks, Section 3 proposes a set of mitigation measures that go beyond organizational controls, presenting a comprehensive framework that integrates both technical and procedural safeguards. These include alignment strategies (e.g., RLHF, Constitutional AI), robust input/output guardrails, hybrid AI-human workflows with explicit verification protocols, and continuous validation using domain-specific benchmarks. To ground up the discussion, Section 4 introduces a concrete use case involving our enterprise-grade AI platform (VIO) for automotive safety engineering, with a particular focus on a MISRA compliance assistant. We present preliminary observations from a pilot user study that suggest both productivity improvements and critical limitations (such as overconfidence and context gaps), along with the specific mitigation measures applied in practice. We conclude that responsible LLM deployment in safety-critical systems requires moving beyond generic guidelines to implement measurable safeguards, maintain human expertise in critical decision-making, and establish continuous validation frameworks that ensure outputs remain within acceptable risk levels as defined by safety engineering standards.
No takes yet. Share an insight, caveat, or question.
Bonnin et al. (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: