ThreatLens: Claim-Level Grounding Verification for LLM-Extracted Cyber Threat Intelligence

Large language models are increasingly proposed for automating cyber threat intelligence (CTI) extraction, but existing benchmarks evaluate them with precision-style accuracy metrics against static gold labels. I argue this measurement approach is unsafe for operational deployment, and I present ThreatLens, an open-source pipeline that performs deterministic, claim-level grounding verification of LLM-extracted CTI against three authorities: the National Vulnerability Database, MITRE ATT&CK STIX data, and the source document itself. Every atomic claim (CVE, indicator of compromise, ATT&CK technique, threat actor, malware family) receives one of five verdicts under a taxonomy that separates fabricated entities from entities that are real but ungrounded in the source, mislabeled metadata, and claims that are simply unverifiable because no authority can confirm them. I evaluate five open-weight models on 94 CISA advisories published after the models' training cutoffs, plus two indicator-list CTI events. Four findings emerge. (i) Hallucination is task-dependent rather than model-intrinsic: Nemotron-3-Super 120B scores 0.00% [95% CI: 0.00%, 0.52%] on indicator-copying tasks and 6.39% [4.65%, 8.73%] on narrative advisories requiring inference. (ii) Precision metrics conceal silent failure: Nemotron-3-nano 30B returned well-formed but empty extractions on 100.0% [72.2%, 100.0%] of its successful runs over CVE-bearing advisories, which a precision-only benchmark would report as flawless. Even Nemotron-3-Super 120B silently returns nothing on 11.2% [7.0%, 17.4%] of CVE-bearing advisories. (iii) Two reproducible failure modes appear at temperature 0: parametric-memory CVE imports and systematic ATT&CK sub-technique confabulation. (iv) Following NIST's April 2026 decision to abandon universal NVD enrichment, existence checking against vulnerability registries fails precisely on the freshest advisories: 9 of 11 CVEs cited verbatim in same-week ICS advisories were absent from both NVD and CVE.org. I further report the verifier's own audit trail: five classes of verifier false positive discovered by manual adjudication and fixed under regression test, which I argue should be a standard reporting requirement for automated CTI evaluation. Code, corpus manifests, and per-claim verdicts are released.