Detecting FIMI: A methodological framework for testing OSINT tools in TTPs detection

<ns3:p>Background Foreign Information Manipulation and Interference (FIMI) represents a growing hybrid threat targeting democratic processes, social cohesion, and diaspora communities. Detecting such operations requires identifying underlying tactics, techniques, and procedures (TTPs) described in frameworks such as DISARM, AMITT, and MITRE ATT&CK. Although numerous open-source intelligence (OSINT) tools exist, their evaluation remains fragmented and lacks methodological standardisation. This article addresses this gap by presenting a structured framework for testing OSINT tools for TTP detection, developed within the Horizon Europe project RESONANT. Methods The methodology integrates three components: (1) a systematic multi-source collection of OSINT tools informed by academic, operational, and grey literature; (2) the creation of synthetic datasets representing text-based, multimedia, network, and infrastructure-level manipulation techniques derived from established FIMI taxonomies; and (3) controlled testing conducted by law enforcement partners from Greece and Ukraine. Tools were classified into eight categories and evaluated using predefined indicators covering accuracy, usability, interoperability, coverage of adversarial behaviours, and sustainability. Results The framework enabled reproducible, ethically compliant testing across tool categories. Findings showed strong performance in content-verification tools (fact-checking and multimedia forensics) and infrastructure-level verification (IP, email, and phone lookups). Social media analytics tools were effective in identifying coordination patterns but varied significantly in usability and exportability. Cross-cutting shortcomings included limited multilingual processing, poor cross-platform interoperability, and difficulties integrating outputs into structured threat-intelligence formats. Tool performance was also affected by API volatility, software updates, and platform policy changes. Conclusions The study provides a reproducible and operationally grounded methodology for evaluating OSINT tools in the context of FIMI TTP detection. By combining synthetic datasets, structured testing, and practitioner validation, it strengthens methodological transparency and supports the optimisation of tool suites for researchers and law enforcement agencies.</ns3:p>