Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers

Recurrent neural networks (RNNs), temporal convolutions, and neural equations (NDEs) are popular families of deep learning models for-series data, each with unique strengths and tradeoffs in modeling power computational efficiency. We introduce a simple sequence model inspired by systems that generalizes these approaches while addressing their. The Linear State-Space Layer (LSSL) maps a sequence u \ y simply simulating a linear continuous-time state-space representatioṅ = Ax + Bu, y = Cx + Du. Theoretically, we show that LSSL models are related to the three aforementioned families of models and inherit strengths. For example, they generalize convolutions to continuous-time, common RNN heuristics, and share features of NDEs such as time-scale. We then incorporate and generalize recent theory on continuous-time to introduce a trainable subset of structured matrices A that LSSLs with long-range memory. Empirically, stacking LSSL layers into a deep neural network obtains state-of-the-art results across time series for long dependencies in sequential image classification, real-world regression tasks, and speech. On a difficult speech classification with length-16000 sequences, LSSL outperforms prior approaches by 24 points, and even outperforms baselines that use hand-crafted features 100x shorter sequences.