APKOS: Accurate and Privacy-Preserving Keyword Spotting for Intelligent Voice Assistant Systems

Intelligent voice assistants and their supporting Cloud Voice Assistant (CVA) systems are becoming increasingly prevalent in AIoT environments. However, transmitting voice data to the cloud raises significant privacy concerns. Homomorphic encryption (HE) offers a promising solution by enabling secure computation over encrypted data without revealing the plaintext. Yet, its application introduces performance challenges for keyword spotting (KWS). Therefore, designing a privacy-preserving KWS scheme that ensures both security and high accuracy remains a significant challenge. In this paper, we propose a privacy-preserving and accurate keyword spotting scheme tailored for CVA systems. Our design is based on HE and achieves semantic security while maintaining high KWS performance. To optimize encrypted inference, we introduce a fusion layer that combines activation and pooling operations, reducing ciphertext-level computation. We also design a real-valued polynomial activation function tailored for audio-based tasks and employ a kernel expansion strategy to enhance representational capacity without increasing multiplicative depth. Extensive evaluations across multiple datasets demonstrate that our method achieves up to 7% improvement in accuracy and 10% gain in robustness compared to state-of-the-art privacy-preserving KWS methods.