Web Crawling.¶
Olston, C., & Najork, M. (2010). Web Crawling. Foundations and Trends in Information Retrieval, 4(3), 175-246.
Cited by¶
1 citation across 1 artifact.
Each citation links to the sentence it supports in the citing article.
Domain-specific¶
- Internet Bot
- RFC 9309 makes these roles explicit for crawlers by standardizing how an automatic client identifies itself and interprets `robots.txt` access rules. Web-crawling research supplies frontier, prioritization, freshness, scale, and politeness mechanisms.
This sourceSurvey of crawler architecture, frontiers, ordering, scale, freshness, and politeness.
- RFC 9309 makes these roles explicit for crawlers by standardizing how an automatic client identifies itself and interprets `robots.txt` access rules. Web-crawling research supplies frontier, prioritization, freshness, scale, and politeness mechanisms.
Verification¶
This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.
Registry ID ref:59e7c66c00d3 · see in the full table