Cloudflare Just Ended AI's Free Ride on the Web
Memeburn, Monday, July 6th, 2026
Cloudflare will block AI crawlers on ad-supported pages by default and require payment for access.
Cloudflare announced a policy, effective September 15, 2026, that defaults to blocking AI training and agent crawlers on any page running ads, while still allowing search crawlers; its data puts mixed-use crawlers at 36% of all crawler activity.
The company is also shifting from its Pay Per Crawl marketplace toward a Pay Per Use model, with payments running on the open x402 protocol backed by Cloudflare, Stripe, Amazon, Google, and Microsoft.
The move aims to stop AI companies from treating public web data as a free input, forcing them to sign real deals for the sources they want and follow each site's crawler rules. (Page returned HTTP 403; details recovered via web search.)