Measuring Semantic Similarity of Persian Sentences Using ParsBERT Model

Semantic similarity is a subset of natural language processing in artificial intelligence that examines the similarity between two texts in terms of meaning. It has various applications, such as plagiarism detection, document similarity checks, encyclopedias, question-answering systems, search engines, and chatbots. To address the need for updated semantic similarity models, we used the ParsBERT language model as a base model. It is trained on Persian data and converts tokenized sentences into meaningful vectors. Through pooling layers, optimized vectors are extracted, resulting in 768-dimensional vectors for each sentence. Cosine similarity is then used to calculate the similarity between the two vectors. The results show that the proposed model achieved a Pearson correlation coefficient of approximately 0.82, surpassing previous models. The best coefficient achieved by previous implementations was around 0.77, using a combination of FastText text embeddings and a CNN model. The proposed model's higher coefficient indicates its superior performance compared to previous models.