What question did this study set out to answer?

To evaluate the feasibility and performance of training deep neural networks using limited numerical precision and low-precision fixed-point computation.

February 9, 20151,079 citationsOpen Access

Deep Learning with Limited Numerical Precision

SGSuyog Gupta AAAnkur Agrawal KGKailash Gopalakrishnan

Key Points

To evaluate the feasibility and performance of training deep neural networks using limited numerical precision and low-precision fixed-point computation.
Trained deep neural networks using 16-bit fixed-point numerical representations.
Investigated the effect of different numerical rounding schemes, specifically testing stochastic rounding against conventional methods.
Designed and demonstrated a dedicated hardware accelerator executing low-precision fixed-point arithmetic with stochastic rounding.
Deep neural networks trained with 16-bit fixed-point representation and stochastic rounding experienced little to no degradation in classification accuracy compared to standard full precision.
The numerical rounding scheme was identified as a critical determinant of network stability and learning behavior during low-precision training.
Hardware implementation demonstrated that low-precision arithmetic with stochastic rounding enables energy-efficient acceleration.

Abstract

Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network's behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithmetic with stochastic rounding.

Ask AI

Helpful

Bookmark

View Full Paper

Cite This Study

Gupta et al. (2015) studied this question.

synapsesocial.com/papers/6a0ac78e7e716524c8aca031 https://doi.org/10.48550/arxiv.1502.02551

Ask AI

Helpful

Bookmark

View Full Paper

February 9, 20151,079 citationsOpen Access

Deep Learning with Limited Numerical Precision

SGSuyog Gupta AAAnkur Agrawal KGKailash Gopalakrishnan

Key Points

To evaluate the feasibility and performance of training deep neural networks using limited numerical precision and low-precision fixed-point computation.
Trained deep neural networks using 16-bit fixed-point numerical representations.
Investigated the effect of different numerical rounding schemes, specifically testing stochastic rounding against conventional methods.
Designed and demonstrated a dedicated hardware accelerator executing low-precision fixed-point arithmetic with stochastic rounding.
Deep neural networks trained with 16-bit fixed-point representation and stochastic rounding experienced little to no degradation in classification accuracy compared to standard full precision.
The numerical rounding scheme was identified as a critical determinant of network stability and learning behavior during low-precision training.
Hardware implementation demonstrated that low-precision arithmetic with stochastic rounding enables energy-efficient acceleration.

Abstract

Ask AI

Helpful

Bookmark

View Full Paper

Cite This Study

Gupta et al. (2015) studied this question.

synapsesocial.com/papers/6a0ac78e7e716524c8aca031 https://doi.org/10.48550/arxiv.1502.02551

Ask AI

Helpful

Bookmark

View Full Paper