Research on Short Text Clustering Algorithm Combining BERT and BTM

In today's era of online globalization, social media has become an important channel for obtaining information. However, the problems of information overload and data sparsity in its short text data bring great challenges to effective information processing. This paper proposes a short text vector representation method based on BERT and BTM topic models, which can not only mine the deep semantic information of short texts, but also effectively overcome the problem of text feature sparsity and enhance the semantic and topic representation of short texts. In order to verify the effectiveness of the vector representation method proposed in this paper, the DBSCAN clustering algorithm was used to cluster the short text. The experimental results show that in terms of clustering performance, the model combined with BERT and BTM has extremely high contour coefficient and CH value, as well as very few noise points. These indicators reflect the effectiveness of the proposed method in terms of the accuracy of clustering and the quality of clustering, which provides a new technical means for short text processing in social media.