A cognitive platform for collecting cyber threat intelligence and real-time detection using cloud computing

The extraction of cyber threat intelligence (CTI) from open sources is a rapidly expanding defensive strategy that enhances the resilience of both Information Technology (IT) and Operational Technology (OT) environments against large-scale cyber-attacks. However, for most organizations, collecting actionable CTI remains both a technical bottleneck and a black box. While previous research has focused on improving individual components of the extraction process, the community lacks open-source platforms for deploying streaming CTI data pipelines in the wild. This study proposes an efficient platform capable of processing compute-intensive data pipelines, based on cloud computing, for real-time detection, collection, and sharing of CTI from various online sources. We developed a prototype platform (TSTEM) with a containerized microservice architecture that uses Tweepy, Scrapy, Terraform, Elasticsearch, Logstash, and Kibana (ELK), Kafka, and Machine Learning Operations (MLOps) to autonomously search, extract, and index indicators of compromise (IOCs) in the wild. Moreover, the provisioning, monitoring, and management of the platform are achieved through infrastructure as code (IaC). Custom focus-crawlers collect web content, processed by a first-level classifier to identify potential IOCs. Relevant content advances to a second level for further examination. State-of-the-art natural language processing (NLP) models are used for classification and entity extraction, enhancing the IOC extraction methodology. Our results indicate these models exhibit high accuracy (exceeding 98%) in classification and extraction tasks, achieving this performance within less than a minute. The system’s effectiveness is due to a finely-tuned IOC extraction method that operates at multiple stages, ensuring precise identification with low false positives. • Use containerized microservice architecture for real-time cyber threat intelligence detection from online sources. • Implement custom focus crawlers to efficiently and accurately extract indicators of compromise (IOCs). • Propose an innovative method for autonomous search and extraction of IOCs using transformer-based algorithms. • Use infrastructure as a code for enhancing the automation of the cyber threat intelligence collection process. • Demonstrate scalable and efficient cloud-based data pipelines for handling computationally demanding tasks.