Answers are grounded in this paper and cite their sources.
Vanilla deep neural networks (DNNs) after training are represented with native floating-point 32 (fp32) weights. We observe that the bit-level sparsity of these weights is very abundant in the mantissa and the distribution of exponent is aggregated, which can all be directly exploited to speed up model inference. In this article, we propose Mortar and Mortar-FP8, the offline/online software and hardware collaborative approaches for fp32 DNN acceleration. The proposed methods include the software algorithms to morph the mantissa and convert fp32 weights to fp8 format, as well as associated hardware accelerator architecture to accelerate general-purpose deep learning through optimized algorithm and specialized hardware. We highlight the following results by evaluating various deep learning tasks, including image classification, object detection, video understanding, video, and image super-resolution: 1) Mortar increase mantissa sparsity up to1.58× -.09×with only a negligible ~0.2% accuracy loss; 2) Mortar-FP8 morph the fp32 weights to fp8 format with a minimal accuracy loss of ~0.3%; and 3) the corresponding hardware accelerator significantly outperforms baselines, achieving up to6.032×and6.99×performance improvements. The area and power of Mortar are 0.031 mm2 and 68.58 mW. Those metrics are 0.0505 mm2 and 25.16 mW for Mortar-FP8.
No takes yet. Share an insight, caveat, or question.
Lü et al. (2023) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: