Context-Aware Vocabulary Pruning for LLM Inference Acceleration

Key Points

• To investigate and implement a method for pruning vocabulary during LLM decoding to enhance inference speed without altering model weights.
• Developed a Dual-Encoder Router for vocabulary pruning
• Applied pruning at each decoding step of the LLM
• Reduced costs associated with matrix multiplication in the lm_head
• Significantly decreased computational costs during inference
• Maintained model performance without weight modifications
• Achieved faster inference times with the new pruning approach