Answers are grounded in this paper and cite their sources.
Demonstrates improved logit computation efficiency in large language models, suggesting significant speed enhancements.
Autoregressive large language model (LLM) inference multiplies the final hidden state against the entire output embedding matrix at every generation step, producing a logit vector of size V of which over 99.99% of entries are discarded. For a model with V = 256,000 and hidden dimension d = 12,288, this is approximately 3 × 10^9 FLOPs per token. The shape of this computation has not changed since the original Transformer. This paper presents LogitFactory, a training-free method that prunes the vocabulary before logit computation using PCA-based cluster alignment and dual-directional filtering. The method achieves 96–100% top-1 accuracy on GPT-2 variants while reducing logit FLOPs by 70–99%. Two implementation choices are essential. First, cosine similarity between the hidden state and vocabulary cluster centroids yields 0% top-1 accuracy on all tested configurations; replacing this with dot-product comparison on PCA-clustered centroids in the original embedding space recovers near-full accuracy. Second, a sample-first execution order runs at 0.09× full-matmul speed; inverting to cluster-first execution raises this to 0.86× (9.6× process-level improvement). Using cluster-first execution in pure PyTorch, the method reaches 1.09× wall-clock speedup at V = 256,000, d = 12,288 (GPT-4 class), 1.97× at V = 500,000, and 2.59× at V = 500,000, d = 16,384. The speedup ratio grows monotonically with V and d.
No takes yet. Share an insight, caveat, or question.
Myeong Jun Jo (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: