Multimodal English corpus text recognition based on unsupervised domain adaptation

With the explosion of multimodal data, it is an important challenge to effectively utilise unlabeled data for cross-modal text recognition. This paper first preprocesses the text and speech data in the English corpus, and use BiLSTM and self-attention mechanism (SA) to extract important text features; and use convolutional neural network, BiLSTM and SA to extract speech features with high contribution. Subsequently, the multimodal features are modelled by graph neural networks, a two-part graph is constructed and knowledge transfer is performed, and domain-invariant features containing information about inter-domain interactions are extracted. Reducing the difficulty of domain adaptation with large inter-domain differences through unsupervised domain adaptation makes the adversarial training process smoother. Finally, the recognition results are obtained by the inference of domain invariant features by the classifier. Experimental results show that the weighted accuracy of the proposed model reaches 93.67%, which significantly improves the recognition effect.