Googlebot now rendering pages with noindex tags

[Steptoe]

I recently read an interesting post about how Googlebot is treating noindex tagged pages, and have quoted some of it here:

Previously, Google would not pass a page through their Web Rendering Service (WRS) if it had a noindex directive, and indeed this is still documented in the Google Search Central documentation.

Basically that meant that if a page had a noindex directive, JavaScript on that page would not be executed, and the page would not be rendered.

...

However, for a while now, the live testing tools, like the URL Inspection Tool, or the rich results test have gone ahead and rendered the page, noindex or not.

...

It seems that Google is now rendering noindex pages, at least in the context of fetch requests.

The fact that the requests made to the test API endpoint were made with a POST method, and not a GET method gives me more confidence that these requests are being made as part of the rendering process, and not because a URL like string was extracted from the page and speculatively fetched.

Source: https://tamethebots.com/blog-n-bits/noindex-does-not-mean-not-rendered

Though this may not affect many, it has implications on sites that use noindex tags to help to 'massage' their crawl budget. From the data in the post linked, it seems to show that Googlebot will use resources to render pages which in the past a noindex tag would prevent them doing so, past a quick glance at the initial html. This could mean that precious crawl budget might get gobbled up on pages that we don't want ending up in the index anyway, and so it may be better to look at actively blocking Googlebot server side for these pages rather than relying on noindex tags.

 

[4440]

Maybe the crawling of the no-index does not consume crawl budgets.

 

[Steptoe]

Maybe the crawling of the no-index does not consume crawl budgets.

If Google is using extra resources to crawl these noindex pages AND render javascript on them (as the test seems to show), I would bet my arse that it will be taken out of the crawl budget. This is not going to affect most, but for those of us working on bigger sites and trying to reduce load whilst keeping crawling healthy, I thought it would be worth giving a heads up.

 

[Panther28]

I was getting the perception from google that they now didn't care about a lot of stuff like this, and they preferred to just let their AI run rampant, and collect every piece of data from pretty much every where they can, screw ethics or legality! Google shouldn't be at the same time setting trends that the web uses, then break the rules of HTML/JS just because they can, and make their life ultimately easier? They might say that malicious code or something can be hidden behind these tags to do the dirty probably. lol

 

[FindYourHost]

I recently read an interesting post about how Googlebot is treating noindex tagged pages, and have quoted some of it here:

Source: https://tamethebots.com/blog-n-bits/noindex-does-not-mean-not-rendered

Though this may not affect many, it has implications on sites that use noindex tags to help to 'massage' their crawl budget. From the data in the post linked, it seems to show that Googlebot will use resources to render pages which in the past a noindex tag would prevent them doing so, past a quick glance at the initial html. This could mean that precious crawl budget might get gobbled up on pages that we don't want ending up in the index anyway, and so it may be better to look at actively blocking Googlebot server side for these pages rather than relying on noindex tags.

You’re right to bring this up.
If Googlebot’s now rendering noindex pages and even running JS on them, it basically kills the old idea of using noindex to control crawl budget.
For big sites, that’s a real concern ,we used to rely on noindex to keep Google from wasting time on stuff we didn’t want indexed.
Now if that processing still happens anyway, we’re burning resources for nothing.
Server-side blocking might be the only clean option left.

 

[4440]

If Google is using extra resources to crawl these noindex pages AND render javascript on them (as the test seems to show), I would bet my arse that it will be taken out of the crawl budget. This is not going to affect most, but for those of us working on bigger sites and trying to reduce load whilst keeping crawling healthy, I thought it would be worth giving a heads up.

I trust your arse!

 

[hideath]

The whole idea about "noindex" is to tell the bots not to index these pages,
They can still be crawled, pass link juice to other internal pages, and some to external pages

Server-side blocking might be the only clean option left.

noindex has never been the way to "stop" Google from crawling the pages,
It's what robots.txt is meant/made for, probably better than server-side blocking, as it will be seen/treated as 503s (you don't want that)

 

[FindYourHost]

The whole idea about "noindex" is to tell the bots not to index these pages,
They can still be crawled, pass link juice to other internal pages, and some to external pages


noindex has never been the way to "stop" Google from crawling the pages,
It's what robots.txt is meant/made for, probably better than server-side blocking, as it will be seen/treated as 503s (you don't want that)

Fair point. The issue with noindex is exactly that, it doesn’t stop crawling, just indexing. But in some edge cases, you do want to stop both, especially for dynamic or sensitive pages that shouldn't be accessed at all.

Robots.txt works, yes, but Google still "tests the fences" sometimes. Server-side blocking is heavier and has downsides (like you said with 503s), but when you need full exclusion, it’s the only thing bots really can’t bypass.

 

[Steptoe]

The whole idea about "noindex" is to tell the bots not to index these pages,
They can still be crawled, pass link juice to other internal pages, and some to external pages


noindex has never been the way to "stop" Google from crawling the pages,
It's what robots.txt is meant/made for, probably better than server-side blocking, as it will be seen/treated as 503s (you don't want that)

The main issue, for me at least, is that Google is meant to not run noindex'd pages it crawls through it's Web Rendering Service:

Source: https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics#use-meta-robots-tags-carefully

But in fact it does, meaning lots of potential resource wastage.

Anyway, thankyou for coming to my TED talk

 

[SEO FIRE]

If a page won’t be indexed, Googlebot may still use crawl resources to render it, wasting crawl budget on pages you don’t want ranked.
What's wrong with you google??
If you want to truly avoid this, block such pages with robots.txt or server-side rules to prevent crawling/rendering altogether.

 

[hideath]

Fair point. The issue with noindex is exactly that, it doesn’t stop crawling, just indexing. But in some edge cases, you do want to stop both, especially for dynamic or sensitive pages that shouldn't be accessed at all.

Robots.txt works, yes, but Google still "tests the fences" sometimes. Server-side blocking is heavier and has downsides (like you said with 503s), but when you need full exclusion, it’s the only thing bots really can’t bypass.

yup, fair point,
Sensitive pages should be blocked for all users except specific IPs or only unlocked when needed (especially if you're managing a big website)

Anyway, thankyou for coming to my TED talk

We need more TED talks for sure

 

[FindYourHost]

yup, fair point,
Sensitive pages should be blocked for all users except specific IPs or only unlocked when needed (especially if you're managing a big website)


We need more TED talks for sure

Yeah, for pages that handle internal logic, admin access, or auth flows, whitelisting IPs or only unlocking access when needed is usually the safest move. Bit more effort, but it shuts out both bots and accidental leaks.

Sometimes locking it down hard is the only thing that actually works long-term, especially when there’s real risk involved.

 

[lightningen]

Really interesting find, thanks for sharing. I used to rely on noindex for controlling crawl budget too, but if Googlebot’s now rendering those pages anyway, it definitely changes things. Blocking at the server level might be the safer bet going forward.

 

[jennykumar]

crawling noindex consumes crawl budget as per john muller from google search.
to avoid this just use user-agent Googlebot disallow in robots.txt so it will not at all go to that page. I've 40K+ tag pages that are noindex,follow. Plus I've added it to robots.txt disallow for bing/googlebot.