Discovering key users in Chinese social networks based on dynamic web crawlers

Weibo, which has over 100 million users, stands as one of China's foremost social media platforms. Among these users, key individuals have become the focal point of numerous studies. However, identifying these key users amid the vast social network presents a formidable challenge. While web crawlers serve as ubiquitous tools for harvesting internet data, the complexity of social network data mining demands specialized algorithms. This paper delves into the analysis of Weibo's webpage structure and content-pushing rules, presenting a method for the discovery of key users utilizing a web crawler. Implementing a web crawler program, developed using Python and Selenium, underscores the practicality of the proposed approach. The results of the experiments validate the effectiveness of the method in extracting data from key users, thereby establishing a database for further research in this domain.