What question did this study set out to answer?

To introduce SVCCA, a technique for analyzing representations in deep learning models.

June 19, 2017Open Access

SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

Key Points

To introduce SVCCA, a technique for analyzing representations in deep learning models.
Developed SVCCA for rapid comparison of different neural network layers and architectures.
Analyzed intrinsic dimensionality and learning dynamics across training.
Suggested new training approaches for improved efficiency.
Identified unnecessary over-parameterization in certain layers.
Showed networks converge to representations in a bottom-up manner.
Recommended new training schedules to reduce computation and overfitting.

Abstract

We propose a new technique, Singular Vector Canonical Correlation Analysis (SVCCA), a tool for quickly comparing two representations in a way that is both to affine transform (allowing comparison between different layers and) and fast to compute (allowing more comparisons to be calculated than previous methods). We deploy this tool to measure the intrinsic of layers, showing in some cases needless over-parameterization; probe learning dynamics throughout training, finding that networks converge final representations from the bottom up; to show where class-specific in networks is formed; and to suggest new training regimes that save computation and overfit less. Code: : //github. com/google/svcca/

AI से पूछें

Bookmark

View Full Paper

Cite This Study

Raghu et al. (Mon,) studied this question.

synapsesocial.com/papers/69d9d9991ad561c6736853f2 https://doi.org/https://doi.org/10.48550/arxiv.1706.05806

AI से पूछें

Bookmark

View Full Paper