Optimization and application of web crawler architecture

Kuncheng Li; Junqi Fei; Chunmei Fan

doi:10.1117/12.2661783

28 December 2022 Optimization and application of web crawler architecture

Kuncheng Li, Junqi Fei, Chunmei Fan

Author Affiliations +

Proceedings Volume 12506, Third International Conference on Computer Science and Communication Technology (ICCSCT 2022); 125060N (2022) https://doi.org/10.1117/12.2661783
Event: International Conference on Computer Science and Communication Technology (ICCSCT 2022), 2022, Beijing, China

Abstract

For monitoring of cutting-edge technologies by obtaining the massive data on the internet, the pyspider framework is used to regularly crawl the information of a large number of websites, and the crawled data is regularly imported into the business application system through python scripts. At the initial stage of the project, the distributed architecture is used as officially recommended by pyspider. Later, it is optimized as the clustered architecture in order to adapt to the actual network environment and the characteristics of crawler tasks. After testing, the work efficiency and stability of the improved architecture have been greatly improved and the expected results have been achieved.

Citation Download Citation

Kuncheng Li, Junqi Fei, and Chunmei Fan "Optimization and application of web crawler architecture", Proc. SPIE 12506, Third International Conference on Computer Science and Communication Technology (ICCSCT 2022), 125060N (28 December 2022); https://doi.org/10.1117/12.2661783

Access the abstract

PROCEEDINGS
5 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Databases

Data communications

Data storage

Internet

Network architectures

Telecommunications

Data centers

Show All Keywords

RELATED CONTENT

40 Gbps optical backbone network deep packet inspection based on...
Proceedings of SPIE (November 24 2014)

Intelligent connected vehicle cloud data platform for park scene
Proceedings of SPIE (December 01 2021)

LoRaWan-Hyperledger robust network integrity on IoT devices
Proceedings of SPIE (May 10 2019)

The unification management of HFC network by using HP Openview
Proceedings of SPIE (December 05 2005)

Analysis and research on the balanced distribution of the network...
Proceedings of SPIE (January 12 2012)

Development of a forestry government agency enterprise GIS system ...
Proceedings of SPIE (November 05 2008)

Server Cache Synchronization Protocol (SCSP) component for directory enabled...
Proceedings of SPIE (November 05 1999)

Subscribe to Digital Library

Receive Erratum Email Alert

Show All Keywords

Keywords/Phrases

Search In:

Publication Years