How Initialization is Related to Deep Neural Networks Generalization Capability: Experimental Study

Deep learning has lately received significant research effort, considering its capabilities followed by the advantages in many domains. The focus of this paper is on improving the generalization ability, which is a key for successful implementation of deep neural networks. In order to answer the question how initialization is related to the deep neural networks' generalization capability, an experimental study is done. The study consists of several experiments, which belong to the data-independent initialization techniques. Using different approaches, the change of generalization gap is presented.