- Sep 21, 2023
- 1,486
- 183
Surprised there hasn't been a thread about this here?
Basically, Cloudflare is accusing Google of using its crawler for both search and AI simultaneously. Cloudflare is giving Google the ultimatum to split the two or get blocked. Started September 15, last week.
Basically, Cloudflare is accusing Google of using its crawler for both search and AI simultaneously. Cloudflare is giving Google the ultimatum to split the two or get blocked. Started September 15, last week.
On September 15, Cloudflare will start cutting off Google traffic for millions of its smallest customers by default.
And those customers won't leave. They've been begging for this.
The story starts with AI Overviews. Google began answering questions directly at the top of search, using content scraped from recipe blogs, how-to sites, and product reviewers, then sending them a fraction of the clicks. One recipe blog called the arrangement a hostage situation. Sites are looking at their traffic charts and realizing the thing they were afraid to lose is already gone.
So this was never the death of the internet. This is publishers going to war with Google, and Cloudflare, which sits in front of roughly 20% of the web, just became their army.
Peel back a layer and Cloudflare's move gets smarter. They aren't technically blocking Google at all. The new rule blocks any crawler that does both search indexing and AI training in one bot. Googlebot is exactly that bot, one front door for both jobs, which is why nobody could ever block the AI half without losing the search half. Cloudflare's message to Google is simple. Split the bot. Keep search separate and we'll allow it by default. Keep the bundle and our network treats you like a scraper.
Which makes this a $107B company dictating terms to a $4.2T one. Cloudflare is 1/40th of Google's size, betting that controlling the pipes beats owning the index.
Now we find out if David can make Goliath blink.