Publishers can now block artificial intelligence training without disappearing from search engine results.
Cloudflare announced a new Accountable designation for artificial intelligence crawling on 15 September 2026. Alongside this framework, the company launched Disallow AI Training, a tool that lets websites refuse AI model training while remaining present in traditional search results.
Mixed-use crawlers gather web data for both search indexes and AI training simultaneously. According to Cloudflare, these mixed bots represent 36.6% of verified crawler traffic across its network. While fewer than 1% of website owners block search crawlers, 17% restrict AI training. Until now, blocking training on mixed crawlers often forced sites to lose search traffic entirely.
Apple, Google, and Microsoft have met Cloudflare's accountability criteria or provided timelines to do so. These standards require operators to support opt-outs for training and AI summaries, provide URL-level usage visibility, and publicly confirm that opting out does not hurt search rankings. Other mixed crawlers will be blocked if an owner chooses to block training.
The company also introduced Bot Preference Sync, which replaces its Managed Robots.txt feature to apply owner crawling rules across supported crawlers automatically. By early 2027, Cloudflare aims to give publishers centralized control over how much content appears in AI summaries.
Newsletter
Markets in your inbox, weekly
LATAM-focused analysis, investing ideas, and the week in finance.
Keep reading
