A Korean Handwritten Text Recognition Method Based on CSWin Transformer and GPT Language Model

HTR is a challenging problem in the field of pattern recognition, as there are many variations in writing style, slope, size, noise, etc., depending on the individual writter’s habits, unlike PTR(Print Text Recognition). In this paper, we design a CSWin transformer-based encoder and a GPT-2 language model based decoder to improve the per-formance of Korean handwritten text recognition. We have improved CSWin transformer based on the self-attention in horizontal and vertical stripes to suit the characteristics of handwritten text recognition, used it as an encoder. The main features of CSWin transformer is the self-attention modules in the horizontal and vertical stripes and local enhancement positional encoding module. The local enhancement positional encoding module can support any image resolution, and is especially advantageous for HTR with line width varying frequently. In this paper, we use the local enhance positional encoding module. We also applied a projection transform to the encoder's features to transform to language-like tokens, and designed a decoder based on the Korean language model. we constructed a Korean subword vocabulary based on the byte pair encoding algorithm. The text corpus used for subword vocabulary construction contains 16,181,377 lines extracted from newspapers, common sense, novels, dialogues, and dictionaries. Since in Korean, too long sequence can be selected into a subword unlike English, so the maximum length of a subword is set to 5 to prevent the length of a subword from being too long when constructing a subword vocabulary. The final subword vocabulary contains 15477 subwords, and we use these subwords as the language tokens. Consequently, we have proposed a Korean handwritten text recognition method consisting of a CSWin transformer-based encoder and a GPT language model-based decoder. Experimental results show that our method is an efficient and effective method for Korean handwriting text recognition.