Cloudflare to block mixed-use AI crawlers on ad-hosting sites from Sept 2026 by default.

Cloudflare announced that starting September 15, 2026, its default settings will block mixed-use web crawlers—those used for both traditional search and AI training—from accessing pages that host ads. This change aims to protect website owners' content from being used freely for AI training without consent, while encouraging AI companies to separate their crawlers by purpose. The update will affect new and existing free Cloudflare customers and could impact how AI models access web data. Cloudflare also offers tools for publishers to monetize AI content use and reduce redundant crawling, partnering initially with Ceramic.ai and You.com to enable payments to content owners when their data is used in AI search results or premium content access.