Evaluating the Effectiveness and Usability of Microsoft Threat Modelling Tool in Undergraduate Cybersecurity Education

As cyber threats evolve, equipping students with hands-on experience in identifying and mitigating system vulnerabilities is critical for developing a cybersecurity-aware workforce. There are a variety of threat modelling tools available on the market, and it is challenging for educators to select the best tool for their students to learn and identify any possible threats that may exploit system vulnerabilities. This study investigates the effectiveness and usability of the Microsoft Threat Modelling Tool (MTMT) among undergraduate students, addressing the need for a practical tool in cybersecurity education. This study was conducted in four phases. The first phase involves conducting a comprehensive literature review to understand the features, capabilities, and tools of the threat modelling tools being compared, specifically the MTMT. Phase two consists of defining the evaluation criteria for assessing the tool's effectiveness and usability. Criteria for error frequency, ease of use, and user-friendliness will be developed, with particular focus on their relevance to educational environments, especially for undergraduate students. Phase three involved data collection, during which participants were recruited and had hands-on sessions with the tool. Training sessions were conducted using case studies to familiarise participants with the tool's features and functionalities. The last phase involves developing assessments to evaluate participants’ knowledge, effectiveness and usability of the tools. The evaluation includes structured usability testing and post-assessment of students’ knowledge and skill acquisition. Findings reveal that MTMT enhances students’ comprehension of threat modelling concepts, bridging the gap between theoretical knowledge and real-world cybersecurity practices. However, the study also highlights areas for improvement in the tool’s interface and documentation to better support student learning. These insights enhance educational strategies, foster active learning, and equip students for real-world cybersecurity challenges. The results emphasise the tool’s potential to strengthen the integration of threat modelling into the cybersecurity field, thereby fostering essential skills for safeguarding organisational and digital infrastructures. The novelty of this study lies in the methodology used to measure the effectiveness and usability of the threat modelling tool. The tool’s effectiveness was measured using the effectiveness formulas from ISO/IEC 25022:2016(E), while its usability was measured using the System Usability Scale (SUS).