Exact Dual-Space Optimization for Neural Networks

Key Points

• This research aims to improve neural network optimization by utilizing a dual space instead of the traditional parameter space.
• Predictive model leveraging M-matrix derived from forward-pass activations to calculate exact gradients.
• Employing matrix-free Conjugate Gradient method for efficient solves of least-squares problems involving weight updates.
• The dual formulation achieves exact optimal weight updates with improved computational efficiency.
• Exact gradients can be computed without a global computational graph, enhancing scalability and performance.