What question did this study set out to answer?

The research aims to challenge the belief that zero initialization degrades neural network learning performance.

March 18, 2026Open Access

Deep Learning with Zero Initialization: Revisiting Symmetry Breaking and Gradient Flow

Puntos clave

The research aims to challenge the belief that zero initialization degrades neural network learning performance.
Introduced a method for learning with zero-initialized weights and biases.
Examined mixed initialization schemes combining zero and random values.
Conducted experiments on various datasets including MNIST and CIFAR.
Utilized multiple architectures like MLPs, CNNs, and ViTs.
Zero initialization can match or outperform random initialization in certain scenarios.
MLPs and CNNs show significant effectiveness with zero initialization.
MLP-Mixers performed comparably even without random parameters.
Results indicate zero initialization can be a valid training method.

Resumen

Abstract Throughout the history of deep learning, artificial neural networks have been widely regarded as incapable of learning when all weights and biases are initialized to zero. Our study challenges this misconception by introducing a method that enables successful learning even when all weights and biases are initialized to zero. Beyond this method, we also examine mixed initialization schemes in which zero and random initialization coexist, showing that learning remains effective even under such partially randomized settings. Experiments on MNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet using multilayer perceptrons (MLPs), convolutional neural networks (CNNs), residual networks (ResNets), vision transformers (ViTs), and multilayer perceptron mixers (MLP-Mixers) show that zero initialization can match or even surpass random initialization in certain scenarios, particularly with MLPs and CNNs. Notably, MLP-Mixers retained comparable performance despite having no randomly initialized parameters. These findings position random initialization as a special case of zero-centered symmetry breaking and refute the longstanding belief that zero initialization inherently degrades neural network performance, opening new possibilities for artificial neural network training. To systematize these insights, we propose the “Seo Integrated Zero Initialization: Foundational Scheme (SIZIFS)” — a unified conceptual structure that categorizes artificial neural network initialization strategies into weight-level, node-level, and context-dependent types. Implementation code is publicly available at: https://github.com/sjw007s/Deep-Learning-with-Zero-Initialization-Revisiting-Symmetry-Breaking-and-Gradient-Flow.

Me gusta

Guardar

Ver artículo completo