Cloudflare will block AI web crawlers by default, and introduces new Pay Per Crawl scheme that means AI companies will have to fork out for the privilege to scrape
As time goes on, just saying ‘no’ to AI feels more and more futile. However, Cloudflare is announcing a few more tools for your anti-AI arsenal that put some of the power back in your hands. Cloudflare offers a range of network services from domain registration and content delivery to DDOS mitigation. Adding to that quiver, Cloudflare is launching the sharp and pointy Pay Per Crawl scheme, which aims to hit AI companies scraping online content where it hurts—namely, their deep pockets.
Last year, Cloudflare began allowing customers to simply block AI bots from their websites with one click. This AI Audit tool also allows users to pick and choose which bots they want to block, as well as offering insights into which LLM or gen-AI is regurgitating their content. Now, Cloudflare has announced that AI crawlers will be blocked by default, describing it as “the first step toward a more sustainable future for both content creators and AI innovators.”
Users who want to have their content scraped for free can still turn this default setting off, but Cloudflare’s second announcement seemingly asks the question, ‘Why would you want to?’
Pay Per Crawl, a scheme announced last year that allows users to charge AI companies wanting to scrape their content, is now in beta. To date, Cloudflare has not revealed the names of any Pay Per Crawl participants aside from ProRata, the startup behind AI search engine Gist.AI.
Founder and CEO Bill Gross offered comment as part of Cloudflare’s announcement regarding AI crawlers, saying, “Our whole mission is to protect and elevate human creativity in the AI era. That’s why we’re proud to be one of the first AI companies to participate in Cloudflare’s initiative to create a new permission-based model for the Internet. We believe that creators and publishers deserve to be fairly compensated for the value they bring, and we’re thrilled to support Cloudflare in helping make that vision a reality.”
It doesn’t seem unlikely that bigger AI companies would go for a scheme like this. After all, OpenAI has already struck content partnerships with a number of media companies, including PC Gamer’s own parent company, Future. Pay Per Crawl feels like an attempt to level the playing field, giving smaller content creators a leg to stand on, who would otherwise simply not have an entire team to dedicate to working out the terms of a content partnership.
But let’s reverse for a second, especially because, as Wired notes, not all web crawlers are bad. For instance, the Internet Archive’s preservation work in particular relies heavily on them. Last year, the Pew Research Center found that a quarter of all webpages that existed between 2013 and 2023 are now no longer accessible. Web crawlers are one way to preserve pages like that before they disappear in a puff of broken links and long-forgotten hosting services.
AI crawlers, however, are a bit of a menace as they scrape without permission from the owner of the content, and can end up overly taxing servers by behaving not unlike a DDOS attack. As previously mentioned, Cloudflare is no stranger to DDOS attacks, telling our Nick last month that their network blocks about ‘225,000,000,000 attacks per day.’ The company also shared that its network recently mitigated the largest attack ever recorded, battering a single IP address with 37.5 TB of data over the course of 45 seconds.
As such, Cloudflare’s AI blocking tools leverage a mix of machine learning and behavioural analysis to differentiate between these two types of web crawlers. Cloudflare has also found some success fingerprinting naughty bots by punting AI web crawlers down a rabbit warren of AI-generated pages. Now that’s what I call fighting fire with fire…or a flinging slop fight. At any rate, would it be naive to feel just a little optimistic about Cloudflare’s push for a ‘Permission-Based Model for the Internet’? Time will tell.
Best gaming PC 2025
