Systematic Study of Different Structural Large Language Models in Semantic Textual Similarity

Semantic Textual Similarity(STS) is one of the challenging problems in the field of natural language processing and plays an important role in many applications such as question answering, recommendation systems and information retrieval. For the STS tasks, previous work has primarily been based on encoder-only structural models, as researchers widely believe that such structure can better capture feature information compared to the other structural models. However, there has been a lack of systematic comparative studies to investigate the performance of different structural models on STS. To fill this gap, we systematically compared different structural models and explored methods for extracting sentence embeddings tailored to different structures and conducted extensive experiments across 7 text similarity datasets. The results show that, the decoder-only model LlaMA2 has shown superior overall performance on the SentEval benchmark without fine-tuning and as the parameters increased, the performance of the decoder-only model improves gradually. Additionally, for any model structure, without doing any projection, the intermediate layers of the model actually degrade the quality of sentence embeddings, affecting the model's performance on STS tasks.