Deep spiking Q-networks for turn-based game environments: encoding choices and energy trade-offs

Deep Reinforcement Learning (DRL) has achieved remarkable success across various domains but at the cost of substantial energy consumption. Spiking Neural Networks (SNNs), inspired by biological neural processing, offer a promising solution through event-driven, sparse computations. To evaluate the capability of SNNs in discrete, strategic decision-making settings, we adopt the Deep Spiking Q-Network (DSQN) framework, whose viability has been established in foundational studies. However, these prior works predominantly restricted DSQN’s application to reactive control tasks and high-frequency visual environments such as Atari games, leaving its efficacy in deliberate, long-horizon strategic reasoning and discrete turn-based environments unexplored. This paper addresses this gap by evaluating the DSQN architecture against traditional Deep Q-Networks (DQN) to investigate whether spike-based value estimation can achieve comparable performance in turn-based strategic tasks. To enable this comparison, we adopt two well-established open-source turn-based environments: Tic-Tac-Toe as a baseline micro-benchmark for turn-based structures, and Connect 4 to assess architectural scalability under complex game environments. Within this framework, we then systematically analyze six distinct SNN input encoding methods: Population, Count Rate, Time-to-First-Spike (TTFS), Rank Order Coding (ROC), Sparse Distributed Representation (SDR), and Burst encoding. Our results demonstrate that DSQNs achieve up to 82.7% and 80.5% detailed analytical energy savings in Tic-Tac-Toe and Connect 4 respectively compared to traditional DQNs, while maintaining comparable strategic performance. In complex environments, a clear trade-off emerges: Population encoding provides the highest strategic success rate (92.80% ± 2.28% win+draw in Connect 4), TTFS offers the maximum energy efficiency at the cost of a reduced win+draw rate (84.80% ± 1.30%), and ROC emerges as an optimal balanced choice, preserving a competitive win+draw rate of 91.80% ± 3.03 while delivering 80.1% detailed energy savings in Connect 4. These findings offer practical insights into selecting SNN encoding strategies tailored to accuracy, and energy constraints.