Cloudflare threatens to cut off Googlebot (search indexing)

atlas_http

Supreme Member
Jr. VIP
Joined
Sep 21, 2023
Messages
1,486
Reaction score
183
Surprised there hasn't been a thread about this here?

Basically, Cloudflare is accusing Google of using its crawler for both search and AI simultaneously. Cloudflare is giving Google the ultimatum to split the two or get blocked. Started September 15, last week.

On September 15, Cloudflare will start cutting off Google traffic for millions of its smallest customers by default.

And those customers won't leave. They've been begging for this.

The story starts with AI Overviews. Google began answering questions directly at the top of search, using content scraped from recipe blogs, how-to sites, and product reviewers, then sending them a fraction of the clicks. One recipe blog called the arrangement a hostage situation. Sites are looking at their traffic charts and realizing the thing they were afraid to lose is already gone.

So this was never the death of the internet. This is publishers going to war with Google, and Cloudflare, which sits in front of roughly 20% of the web, just became their army.

Peel back a layer and Cloudflare's move gets smarter. They aren't technically blocking Google at all. The new rule blocks any crawler that does both search indexing and AI training in one bot. Googlebot is exactly that bot, one front door for both jobs, which is why nobody could ever block the AI half without losing the search half. Cloudflare's message to Google is simple. Split the bot. Keep search separate and we'll allow it by default. Keep the bundle and our network treats you like a scraper.

Which makes this a $107B company dictating terms to a $4.2T one. Cloudflare is 1/40th of Google's size, betting that controlling the pipes beats owning the index.

Now we find out if David can make Goliath blink.

 
Surprised there hasn't been a thread about this here?

Basically, Cloudflare is accusing Google of using its crawler for both search and AI simultaneously. Cloudflare is giving Google the ultimatum to split the two or get blocked. Started September 15, last week.



but in any case, I would definitely keep my eye on this issue, yet, since googlebot will be enabled by default, there won’t be much change for the websites concerning indexing. It is only at this point that the real fun start.
 
Cloudflare wants Google to separate its search crawler from its AI training bot, or they’ll block the AI one
 
mine were all on default allow, worth checking the bot setting on each site though. i have a few small support sites on the free plan and one had a rule from an old setup blocking everything but humans. that one had stopped indexing for a month and i only noticed because of this thread
 
Back
Top