Answers are grounded in this paper and cite their sources.
This article is devoted to the technologies of web scraping (web crawling) for Node.js, used in the task of aggregating information and estimating the parameters of cargo routes by extracting data from open sources. The challenge of web scraping occurs in many different contexts, both scientific and industrial. The tasks of web scraping have both wide practical applications and a significant educational aspect. However, the existing material on web scraping is scattered and unstructured. In this paper, using the example of solving the scientific and technical problem of aggregating information and evaluating the parameters of cargo routes by extracting data from open sources, an overview of the technologies for web scraping on Node.js is presented, the classification of sites by complexity is described, the systematization of the features of sites that are an obstacle to scrapinf is given, and possible ways to bypass them. Thus, the didactic goal of this article is achieved to systematize the material on parsing websites.
No takes yet. Share an insight, caveat, or question.
Korepanova et al. (2021) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: