Over the past year, major news publishers have taken aggressive anti-scraping measures to curb the rise of AI crawlers. So far, most of these backend interventions have been out of reach for independent journalists who publish on third-party newsletter platforms.
That is changing with a new partnership announced on Tuesday between Beehiiv and Cloudflare, the internet infrastructure company that handles roughly 20% of all web traffic. Now, all Beehiiv creators will have beta access to Cloudflare’s AI Crawl Control services through their dashboards. These tools allow creators to opt in or out of AI agents crawling their work.
Each creator on Beehiv will see granular information about which AI agents are accessing their newsletters, which ones are being blocked, and how much referral traffic is being sent back to them by associated AI products. Creators will also be able to allow or block access for each of those agents. Cloudflare will automatically update these settings to reflect any new agents that hit the web from the same companies.
Among newsletter publishing platforms, this gives Beehiiv creators some of the most on-platform control over how their content is being used by AI products. Substack offers a setting that disallows scraping via robots.txt files. This signals to AI crawlers that they’re not wanted, but effectively operates on an honor-code system and doesn’t actually block those crawlers. Creators on Ghost have used similar robots.txt file customizations to minimize scraping, according to user forums.
The Beehiiv partnership is the latest move in Cloudflare’s campaign to give publishers more control over how their work is being ingested by AI models — and to build new business around those controls. Last year, Cloudflare made blocking AI crawlers the default for all of its customers and launched a “pay-per-crawl” marketplace, which allows Cloudflare customers to charge an AI crawler a small fee each time it accesses its content. Cloudflare is estimated to be taking a 30% cut of publisher earnings from the marketplace. (There’s no indication that Beehiiv creators will have direct access to the pay-per-crawl marketplace through the new partnership.)
“As AI changes how people find and consume content, publishers need real leverage,” Tyler Denk, the co-founder and CEO of Beehiiv, said in a statement. “Our partnership with Cloudflare gives creators the data and controls they need to either maximize discovery and distribution, or protect their writing and dictate their own terms.”
