Robots.txt is not access control

Robots.txt was built three decades ago as an informal handshake for search crawlers, not an access control system. As model development multiplied across the industry, site administrators found themselves maintaining brittle blocklists, adding separate disallow rules every time a new company launched its own data scraper. To reduce that overhead, OpenAI plans to honor a shared signal for AI training preferences across participating providers, letting website operators set rules without managing each user agent individually.

There is also an obvious irony in OpenAI championing this effort. Commercial labs built their foundation models by running roughshod over data access rights across the open web, making this push for polite opt-out signals look like pulling up the ladder behind them. It also highlights an overlooked dilemma for site maintainers. The web was built for public consumption by people, but AI training introduces new conflicts over licensing and data access. How do you keep content freely accessible to human readers without automatically feeding it into model pipelines? Captcha adds unwanted friction for many site maintainers and as models increase in capability it becomes trivial automate.

A shared directive cuts down the administrative drag of chasing individual crawlers, which helps standardize how publishers deal with commercial labs. The operational reality, however, is that robots.txt remains an honor system. A cooperative vendor can agree to respect the flag, but the protocol provides no technical barrier against scrapers, bulk data extractors, or developers operating outside the agreement. Signaling preferences establishes a clear boundary for responsible partners, but anyone handling proprietary content will still need rate limits, bot mitigation, and authentication at the network edge.