Randomized trial demonstrates compute savings in transformer inference by applying a dynamic exit criterion, indicating potential for efficiency gains.
Key Points
This research aims to enhance early exit methods in transformer models by introducing a dynamic power metric as an exit criterion.
Simulation study calibrated to BERT-base architecture with 12 layers and 600 inputs across four difficulty tiers.
Application of the stochastic power metric P(t) for determining early exits based on dynamic confidence levels.
Performance comparison with existing early exit methods including confidence thresholding and the patience mechanism.
Power metric achieved 55.9% compute savings with 99.7% accuracy preservation.
For confidence threshold method: 14.6% compute savings at 100% accuracy.
For patience mechanism: 52.6% compute savings at 100% accuracy.