Factorization tricks for LSTM networks

Oleksii Kuchaiev; Boris Ginsburg

doi:10.48550/arxiv.1703.10722

March 31, 2017Open Access

Factorization tricks for LSTM networks

View Full Paper

Ask AI

Bookmark

Share

Key Points

This research aims to optimize the training process of large LSTM networks by reducing parameters and enhancing speed.
Implemented matrix factorization by design to break down LSTM matrices into smaller components.
Partitioned LSTM matrices, inputs, and states into independent groups to improve processing efficiency.
Achieved significant reduction in training time while nearing state-of-the-art perplexity.
Utilized fewer RNN parameters compared to traditional LSTM configurations.

Ask about this paper

Answers are grounded in this paper and cite their sources.

Implication

Demonstrates improved training speed in LSTM networks, implying efficiency in deep learning models.

Abstract

We present two simple ways of reducing the number of parameters and accelerating the training of large Long Short-Term Memory (LSTM) networks: the first one is "matrix factorization by design" of LSTM matrix into the product of two smaller matrices, and the second one is partitioning of LSTM matrix, its inputs and states into the independent groups. Both approaches allow us to train large LSTM networks significantly faster to the near state-of the art perplexity while using significantly less RNN parameters.

Discussion

No takes yet. Share an insight, caveat, or question.

Cite This Study

Kuchaiev et al. (2017) studied this question.

synapsesocial.com/papers/6a09638987ad1657d2513fda https://doi.org/10.48550/arxiv.1703.10722

View Full Paper

Ask AI

Bookmark

Share

Factorization tricks for LSTM networks

Key Points

Ask about this paper

Implication

Abstract

Discussion

Cite This Study

Factorization tricks for LSTM networks

Key Points

Ask about this paper

Implication

Abstract

Discussion

Cite This Study

Keep reading

Keep reading