December 20, 2013Open Access
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Authors
Discussion
Loading...
Member takes
Implication
Randomized trial investigates learning dynamics, revealing depth-independent conditions for effective training.
Key Points
- This research aims to understand the nonlinear dynamics of learning in deep linear neural networks and find exact solutions to these dynamics.
- Analyzed the learning dynamics of deep linear neural networks with a specific focus on gradient descent behavior.
- Identified conditions under which learning speed remains finite despite increasing network depth.
- Explored initial conditions for weights that facilitate depth-independent learning times.
- Deep linear networks show nonlinear learning patterns similar to nonlinear networks, including long plateaus in error reduction.
- Found that certain initial conditions, achieved via unsupervised pretraining, lead to faster learning speeds than random initializations.
- Developed a class of random orthogonal initial conditions that support effective learning in both deep linear and deep nonlinear networks.
Cite This Study
Saxe et al. (2013) studied this question.