Towards Responsible AI for IoT Network Security Auditing Using Knowledge Graph and RAGAS

The trustworthiness of AI-powered network security auditing depends not only on detection accuracy but on the faithfulness of the explanations that support compliance verdicts. In IoT network security, Large Language Models (LLMs) are increasingly utilized to produce natural-language security assessments from raw network traffic, yet the extent to which these explanations are grounded in retrieved evidence is rarely measured. This paper presents the Retrieval-Augmented Generation Assessment Suite (RAGAS) as an evaluation framework that compares three retrieval paradigms—rule-based heuristic scoring, dense vector retrieval, and knowledge graph traversal—on the task of explaining network compliance against ETSI EN 303 645 IoT cybersecurity provisions. Using 30 human expert-validated compliance scenarios derived from the CIC-IoT2023 dataset and three LLMs (DeepSeek-R1, Qwen-2.5, Llama-3.2), we find that graph-based retrieval achieves the highest faithfulness (0.570), outperforming rule-based (0.524) and vector retrieval (0.509). All methods, however, exhibit low context recall (≤22.4%), and we highlight that high detection F1 scores do not guarantee faithful explanations; over 40% of statements in compliance answers are unsupported by retrieved evidence. A proof-of-concept prototype, Security Audit Compliance Agent (SACA), demonstrates how knowledge graph traversal can be integrated with interactive visualization to support human auditor oversight. We argue that, in adherence to responsible AI principles, faithfulness measurement should become a standard complement to accuracy reporting for an AI-driven network audit or forensic analysis.