Right now, if you don’t want your site to be scraped by AI companies, you have to block each company’s crawlers individually. On Friday, Anthropic, OpenAI, Google, and Microsoft said they would support a common robots.txt signal that, per OpenAI, “makes it easier for publishers and website operators to communicate the same preference to multiple participating providers at once.”
Anthropic engineer Kevin Kelley proposed the common signal as part of work for the Internet Engineering Task Force, and Anthropic is already supporting it. OpenAI says it will do so “in the coming months,” and Google and Microsoft say they’ll do so next year.
Meta VP of communications Andy Stone called the common signal “an important step,” writing in a tweet that the company supports “publishers’ ability to express preferences about how their content is used for foundational generative AI training.” Meta is under scrutiny for proposed commercial terms that would allow broad scraping; last week, it agreed to delay implementation of those terms.
