2.1.1. Multiple Uses for Crawl Data

Questa pagina riassume la sezione corrispondente di RFC 9969 e conserva termini, date e riferimenti RFC essenziali.

Crawl data can have multiple uses when a vendor also operates another product, such as a search engine, or when the crawler is not the AI vendor. Both patterns are common: Internet search operators also train AI models, and many AI models use third-party crawl data.

Participants viewed traditional search crawling as at least partly aligned with publisher interests, because indexing can return traffic and attention. They did not see the same relationship for AI training; some viewed AI as hostile to publishers because it can draw traffic away.

When crawling has multiple uses that include AI, participants expected "collateral damage" for non-AI uses, especially when publishers use stronger controls such as blocking or paywalls. Some worried that this could effectively "lock down the Web" and affect freedom of expression.