The English to Telugu CLIR for NER using Bidirectional Encoding and Unidirectional Decoding using Random Sampling and Beam Search

Neural Machine Translation (NMT) systems and the availability of a wide variety of linguistic resources have greatly improved Cross-Lingual Information Retrieval (CLIR) capabilities. When translating English queries into Indian languages, the NMT approach performs well. The NMT will employ a parallel corpus for translations. The translation of English queries into Telugu is the main emphasis of this study. A lack of Telugu-language content makes it challenging to have a sizable parallel corpus. Consequently, NMT encounters issues with Out-Of-Vocabulary (OOV) and Named Entity Recognition (NER). Byte Pair Encoding (BPE) attempts to translate unusual words by breaking them down into subwords in order to overcome the OOV problem. Problems such as NER still have an effect. The system may be trained in both forward and reverse directions to recognize NER effectively. The system is trained to recognize named entities in both directions through bidirectional encoding. Consequently, NER issues can be solved with Bidirectional Long Short-Term Memory (BiLSTM) encoding. Random sampling and beam search decoding with unidirectional LSTM are used to improve the translation output sequence. The approach using BPE and BiLSTM encoding, along with random sampling and beam search decoding with unidirectional LSTM, will help to resolve the OOV and NER problems and improve the output sequence of the translations generated by the NMT system. This approach is evaluated by using the Bilingual Evaluation Understudy (BLEU) score and other metrics like accuracy, perplexity, and cross-entropy, demonstrating that the translation quality of NMT with bidirectional encoding and unidirectional decoding using random sampling and beam search surpasses that of regular encoding and decoding models using LSTM.