2.3.1. robots.txt (and Similar)

Cette page resume la section correspondante du RFC 9969 et conserve les termes, dates et references RFC essentiels.

The Robots Exclusion Protocol [RFC9309] is widely recognized by AI vendors as an attachment mechanism for preferences, but participants identified several deficiencies.

It does not scale well for large sites where authors want granular policies across different content. It is usually controlled by the site administrator, which may prevent individual creators on large platforms from expressing their own preferences. If content is copied or moved, preferences at the new site must be transferred explicitly because robots.txt is a separate resource.

These issues led many participants to conclude that robots.txt cannot be the only opt-out solution. It should be part of a larger system. Other similar mechanisms have been proposed, but none appeared to have the same level of attention or implementation.