Escalation-Aware Agentic RAG for First-Line IT Incident Triage: Learned Abstention, Semantic Grounding Analysis and Human Escalation

Large language models can produce remediation suggestions for IT incidents, but the evidence behind their suggestions and the technical correctness of their suggestions are unclear. We evaluate a prototype escalation-aware retrieval-augmented generation (RAG) system for supervised first-line incident triage, using IBM TechQA retrieval, a QLoRA specialized Llama-3-8B-Instruct model, explicit abstention, and Jira ticket escalation. The final reranker and decision thresholds were trained or calibrated on TechQA training questions, but previous choices of retrieval models and contexts used to train the adapter depended indirectly on development questions, so the overall development process is not fully leakage-free. On 205 missing-evidence TEST cases, correct abstention increased from 8.5% (Base) to 42.4% (QLoRA), and hazardous answering decreased from 91.5% to 57.6%. The gate had higher correct abstention (53.7% vs. 42.4%; a significant 11.2 percentage-point advantage, Holm p = 0.013) with no detectable difference in false abstention when operating at their own training-calibrated operating points. At retrospectively matched false abstention, QLoRA had an estimated advantage of +11.2 points over the gate. This different, TEST-threshold-selected contrast was exploratory and not statistically significant. In exploratory NLI analysis, 26.1% of citation-valid QLoRA answers were flagged as containing an unsupported step. NLI–human agreement did not meet the pre-specified acceptance criterion (κ = 0.324 < 0.5), so these estimates cannot be interpreted as validated semantic-grounding rates. In the paired Jira simulation, 64.1% of tickets were closed with a proposed resolution, including 54.9% of naturally unanswerable tickets. In an internal assessment by two research-team annotators, an estimated 14.2% of all incidents were automatically closed with fully correct proposed answers (not independently verified repairs) and potentially risky actions appeared in 24.0% (6/25) of sampled automatically closed missing-evidence tickets. Results support supervised triage and human escalation, not safe autonomous technical remediation.