Inverted Alignments for End-to-End Automatic Speech Recognition

In this paper, we propose an inverted alignment approach for sequence classification systems like automatic speech recognition (ASR) that naturally incorporates discriminative, artificial-neural-network-based label distributions. Instead of aligning each input frame to a state label as in the standard hidden Markov model (HMM) derivation, we propose to inversely align each element of an HMM state label sequence to a segment-wise encoding of several consecutive input frames. This enables an integrated discriminative model that can be trained end-to-end from scratch or starting from an existing alignment path. The approach does not assume the usual decomposition into a separate (generative) acoustic model and a language model, and allows for a variety of model assumptions, including statistical variants of attention. Following our initial paper with proof-of-concept experiments on handwriting recognition, the focus of this paper was the investigation of integrated training and an inverted decoding approach, whereas the acoustic modeling still remains largely similar to standard hybrid modeling. We provide experiments on the CHiME-4 noisy ASR task. Our results show that we can reach competitive results with inverted alignment and decoding strategies.