Large Language Models (LLMs) are being increasingly applied in various natural language processing tasks including safety-critical systems (e.g., medical diagnosis querying, and code generation for self-driving), where resilience to hardware transient faults is essential for guaranteed safety. Traditional Fault Injection (FI) approaches are time consuming due to a large number of repeated executions, which limits their scalability to fast evaluate large-scale resilience. To address these challenges, we propose LLM-IARE, a novel Input-Aware Resilience Estimation Model for LLMs under hardware transient faults. It takes advantage of twice static analysis and once dynamic execution to extract critical parameters to compute general resilience metrics such as Silent Data Corruption (SDC) rates. Fast static analysis can obtain the primary LLM parameters, while dynamic execution can provide input-sensitive attention profiling to characterize how input variations influence internal attention patterns dynamically. More importantly, our proposed LLM-IARE uses the obtained parameters for modeling at three levels (operation, module, and layer) so that the SDC rates of transient fault impacts on LLMs can be calculated quickly and accurately. Additionally, LLM-IARE is further extended to estimate the LLM application-level resilience metric, the cosine similarity reflecting the bit-upset induced semantic fault impacts on final output quality. Comprehensive experiments on six representative LLMs (for example, GPT-2, T5 and RoBERTa), and 30 BERT variants demonstrate that LLM-IARE achieves a fast and accurate LLMs resilience evaluation, with up to 7335 × (average 4500 ×) speedup and an average logarithmic relative error of 3.94% compared to advanced LLVM-based fault injection methods. We further extend the evaluation to four larger Qwen2.5 models (0.5B–7B), where LLM-IARE maintains stable accuracy with logarithmic relative errors between 1.96% and 4.21% (average 3.39%).
Jiao et al. (Mon,) studied this question.