What question did this study set out to answer?

This research aims to demonstrate that token selection in large language models can be executed without full matrix multiplication.

May 31, 20260 citationsOpen Access

Token Selection Without Matrix Multiplication: Experimental Evidence for Retrieval-Based Inference

Ver artículo completo

文韩文国韩

Puntos clave

This research aims to demonstrate that token selection in large language models can be executed without full matrix multiplication.
Implemented a retrieval-and-voting mechanism to replace softmax-based token selection
Conducted experiments to measure offline accuracy and token selection rates
Evaluated the efficiency on a consumer-grade Mac Studio at 671 tokens per second.
Achieved 91-100% offline accuracy in token selection
Sustained a 99% non-BOS token rate during inference
Demonstrated significant performance without traditional matrix multiplication methods.

Resumen

We present experimental evidence that token selection in large language models can be performed wit hout computing the full transformer forward pass. By replacing softmax-based token selection with a retrieval-and-voting mechanism, we achieve 91-100% offline accuracy and a sustained 99% non-BOS to ken rate in standalone inference at 671 tokens per second on a consumer-grade Mac Studio. Our resul ts challenge the default assumption that inference quality is inseparable from matrix multiplicatio n.

Preguntar a la IA

Me gusta

Guardar

Ver artículo completo

Cite This Study

文国韩 (Fri,) studied this question.

synapsesocial.com/papers/6a1bd1db5783ba022b6fd378 https://doi.org/https://doi.org/10.5281/zenodo.20437056

Preguntar a la IA

Me gusta

Guardar

Ver artículo completo