What question did this study set out to answer?

This research aims to demonstrate that token selection in large language models can be executed without full matrix multiplication.

May 31, 20260 citationsOpen Access

Token Selection Without Matrix Multiplication: Experimental Evidence for Retrieval-Based Inference

View Full Paper

文韩文国韩

Key Points

This research aims to demonstrate that token selection in large language models can be executed without full matrix multiplication.
Implemented a retrieval-and-voting mechanism to replace softmax-based token selection
Conducted experiments to measure offline accuracy and token selection rates
Evaluated the efficiency on a consumer-grade Mac Studio at 671 tokens per second.
Achieved 91-100% offline accuracy in token selection
Sustained a 99% non-BOS token rate during inference
Demonstrated significant performance without traditional matrix multiplication methods.

Abstract

We present experimental evidence that token selection in large language models can be performed wit hout computing the full transformer forward pass. By replacing softmax-based token selection with a retrieval-and-voting mechanism, we achieve 91-100% offline accuracy and a sustained 99% non-BOS to ken rate in standalone inference at 671 tokens per second on a consumer-grade Mac Studio. Our resul ts challenge the default assumption that inference quality is inseparable from matrix multiplicatio n.

AIに質問

Bookmark

View Full Paper

AIに質問

Bookmark

View Full Paper

Token Selection Without Matrix Multiplication: Experimental Evidence for Retrieval-Based Inference

Key Points

Abstract

Cite This Study