SpAI Phishing: A Design and Evaluation of AI-assisted Large Spear Phishing Cybersecurity Drill

Generative artificial intelligence has increased both the realism and scalability of social-engineering threats, making conventional phishing education exercises less representative of current attack scenarios. Prior work has shown that large language models (LLMs) can generate convincing phishing and spear-phishing emails, but deploying fully individualized LLM-assisted phishing simulations for thousands of recipients remains costly, labor-intensive, and difficult to govern. This study presents SpAI Phishing, a cost-aware and privacy-preserving LLM-assisted framework for large-scale phishing cybersecurity education drills. The framework uses semantic keyword extraction, context-driven categorization, and cluster-level prompting to generate context-aware simulated phishing emails without creating a unique LLM-driven email for each recipient. A same-cohort, simultaneously delivered, two-condition phishing drill was conducted with 4,065 recipients, including 3,611 students and 454 faculty and staff members. Each recipient received two emails at the same scheduled time: one conventional non-context-aware control email and one cluster-based LLM-assisted context-aware email. To protect the privacy of individuals, the system retained only anonymized cohort-level attributes and aggregate open and click outcomes, without retaining participant-level paired open and click records. The results show that in the cohort as a whole, the experimental condition produced higher aggregate open and click-through rates than the control condition. The open rate increased from 11.24% to 29.10%, and the click-through rate increased from 1.50% to 6.67%. Among students, the impact was especially high: the open rate increased from 10.91% to 30.74%, and the click-through rate from 1.41% to 6.92%. Furthermore, mapping the recipients into semantic categories led to a reduction of the LLM email generation calls from 4,065 to 45, corresponding to an approximate 98.89% reduction in LLM email-generation calls under the cluster-level design. These results suggest that semantic clustering of simulated phishing emails is a practical compromise between conventional generic phishing simulations and fully individualized LLM-assisted spear-phishing simulations. This study provides a scalable deployment pipeline and cost model for privacy-preserving institutional phishing drills. The findings should therefore be interpreted as aggregate, privacy-preserving cohort-level evidence rather than as individual-level causal estimates of cybersecurity awareness or behavior.