An Efficient Multi-Threaded Web Crawler Using HashMaps

In last decades, the number of web pages on the Internet has been exposed a rapid increase intrinsically, and the information on the Internet has reached a very large size.Search engines have been developed to access this large-scale information efficiently.Web crawlers play a very important role in search engines.In this paper, an efficient multi-threaded web crawler is proposed, and empirically analyzed in terms of crawling speed and coverage.