What question did this study set out to answer?

The goal was to develop an effective spiking architecture for large language models using consumer GPUs.

May 14, 2026Open Access

CTP-Hybrid: From Hybrid Architecture to Native Spiking Foundation — A Two-Phase Report on Consumer-GPU Spiking LLMs

Key Points

The goal was to develop an effective spiking architecture for large language models using consumer GPUs.
Established Phase 1 with 7B-scale spiking-augmented LLM fine-tuning on an NVIDIA RTX 5060 Laptop.
Developed custom functions for stable gradients to address issues during tri-module training.
Conducted Phase 2 resulting in successful training of both a hybrid spiking model and a pure spiking model.
Achieved WikiText-2 perplexity of 16.58 with fluent Chinese generation in Phase 1.
Successfully performed tri-module joint training on a 7B hybrid model with zero NaN results.
The pure spiking baseline trained from scratch reached a perplexity of 13.57, demonstrating effective language learning without traditional components.

Abstract

We present the complete two-phase development of CTP-Hybrid, a causal-topological pulsing architecture for large language models. Phase 1 established the first reproducible 7B-scale spiking-augmented LLM fine-tuning on an 8 GB consumer GPU (NVIDIA RTX 5060 Laptop), achieving WikiText-2 perplexity of 16.58 with fluent Chinese generation using TD-SCA microcolumns. However, full tri-module joint training (IST+TCP+DSM) failed across seven attempts due to gradient collapse from discrete pulse operations. Phase 2 resolved this fundamentally. We designed three custom autograd.Function implementations—IntervalSpikeEncoder, TemporalCausalMask, and LIFSpikeSurrogate—that provide stable surrogate gradients for previously non-differentiable pulse operations. These functions enabled: (1) the first successful tri-module joint training on a 7B hybrid model with zero NaN, and (2) a fully Transformer-free, 20M-parameter pure spiking baseline trained from scratch on WikiText-2, achieving a perplexity of 13.57. The pure spiking model contains no self-attention, no LayerNorm, and no floating-point FFN, proving that native pulse computation alone can learn natural language. All code, weights, and experiment logs are released.

Read Full Paperexternally

Ask AI

Helpful

Bookmark

View Full Paper