A systematic comparison of Named Entity Recognition approaches for Cyber Threat Intelligence

Given the large volume of cybersecurity information generated daily, cybersecurity analysts require automated tools to efficiently process this data and extract meaningful insights and actionable outcomes. One such concept is Named Entity Recognition (NER). In this work, approaches for NER on Cyber Threat Intelligence (CTI) data are examined through a Systematic Literature Review (SLR) enabled by a large language model (LLM)-based extractor proposed in this study. Furthermore, seven publicly available approaches are evaluated regarding cost and performance. To provide meaningful insights into tagging performance, we analyze it from four perspectives: overall performance, performance on specific entity types, performance on specific entity attributes (e.g. entity frequency) for interpretability and the capability to tag entire documents. The evaluation shows that, on the datasets evaluated, additional architectural complexity does not yield measurably better tagging performance, while it increases training and inference cost substantially. Encoder-based language models (LMs) such as BERT, combined with non-complex decoding layers, already achieve the highest tagging performance. Under the evaluated full-document setting on STIXnet, additional document-level context did not improve performance and degraded it for most models. The analysis further shows that NER performance scores may be influenced by familiarity with entity strings seen during training, caused by entity overlap between training and test data. The evaluated prompt-based LangExtract configurations, which rely on the general knowledge of two locally deployed LLMs, underperform the specialized approaches and incur higher operational costs.