RFC 9309

Koster, M., Illyes, Zeller, & Sassman, L. (2022). RFC 9309: Robots Exclusion Protocol.

Type

Report

Intellectual base

Grey literature

Year

2022

Link

https://www.rfc-editor.org/rfc/rfc9309.html

Cited by

1 citation across 1 artifact.

Each citation links to the sentence it supports in the citing article.

Domain-specific

• Internet Bot
  • RFC 9309 makes these roles explicit for crawlers by standardizing how an automatic client identifies itself and interprets `robots.txt` access rules.

    This sourceIETF Standards Track specification for communicating access rules to automatic clients known as crawlers, including crawler product-token identification and rule matching.

Verification

This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.

Registry ID ref:b397d5895c73 · see in the full table