A structure based on TrOCR transformer and large language model for classification of handwritten texts

Processing handwritten texts and classification and their content analysis are among the most important problems in the realm of text analysis. Microsoft has presented pre-trained TrOCR models for printed and hand-written texts. These models due to prior pre-training are better starting point for image processing. For using TrOCR with the aim of detecting printed and handwritten texts, we can use fine-tuning technique on pre-trained model using different datasets. This process helps the model to learn better the specific features of image processing and hand-written or semihandwritten texts. TrOCR uses transformer models for OCR and its fine-tuning on special datasets especially, hand-written datasets is a common task. TrOCR model from Microsoft extracts text from these images, and in this research a structure based on TrOCR and LLM has been proposed whose aim is extraction of hand-written texts from existing images in a dataset (English handwritten line dataset) and converting them to text data and then this data has been given to LLM as an input so that the extracted texts can be classified (using BART model) based on different subjects and contents.