A Model Decomposition-in-Time of Recurrent Neural Networks: A Feasibility Analysis

In the context of Recurrent Neural Networks, minimization of the Loss Function (LF) causes the most training overhead. Following the Parallel In-Time approaches, we introduce an ab-initio decomposition across time direction. The key point of our approach lies in the innovative defi nition of local objective functions which allows us to overcome the sequential nature of the network and the management of dependencies between time steps. In particular, we defi ne local RNNs by adding a suitable overlapping operator to the local objective functions which guarantees their matching between adjacent subsequences. In this way, we get to a fully parallelizable decomposition of the RNN whose implementation avoids global synchronizations or pipelining. Nearest neighbours communications guarantee the algorithm’s convergence. We hope that these fi ndings encourage readers to further extend the framework according to their specifi c application requirements.