Differentiable Neural Architecture Search for High-Dimensional Time Series Forecasting

This study applies neural architecture search (NAS) techniques to the modeling of high-dimensional time series data such as multi-variate stock indices. It is known that traditional NAS method applies fully connected directed acyclic graph (DAG) for searching cell structures that requires high computational cost and cannot include the two-input operations such as the Hadamard product. To address the drawback of the DAG backbone, a novel two-input backbone cell architecture for recurrent neural networks is proposed, in which each candidate operation is also carried out with two inputs. Instead of using DAG, we simplify the backbone by considering the prior knowledge as an effective backbone such as preserving identity mappings. The cell structures will be incorporated in different types of model architectures including stacked long short-term memory (LSTM), gated recurrent unit (GRU) and attention-based encoder-decoder models. The experimental results on BRICS, G7 and G20 indices indicate that models with recurrent neural network (RNN) cells searched by the proposed backbone structure can significantly outperform baseline models including autoregressive integrated moving average model (ARIMA), vector autoregression (VAR), and stacked LSTM/GRUs. For neural architecture search, the proposed backbone is shown to be more effective compared to the classic differentiable architecture search (DARTS) in both uni-variate and multi-variate time series prediction tasks. Further analysis demonstrates that the pruned cells of the proposed backbone usually contains the Hadamard product introduced as a two-input operation, while the number of parameters involved in these pruned cells is on the same order with the baseline cells.