Google webmasters - Why is Indexing Allowed? Yes

The webpage has this tag:


The robots.txt has this in it:


When I run the webpage in Google inspection tool, I get this:



We have "Indexing allowed? Yes"

Why?

Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why?

it should be noindex, nofollow

<meta name="robots" content="noindex, nofollow">

shinumunshi said:

it should be noindex, nofollow

<meta name="robots" content="noindex, nofollow">

Click to expand...

I actually disagree. I want Google to follow the links on the page and whatnot, I just don't want google to index the actual page.

@BlogPro

Scorpion Ghost said:

I actually disagree. I want Google to follow the links on the page and whatnot, I just don't want google to index the actual page.

@BlogPro

Click to expand...

You are correct.

Adding nofollow to a meta tag will prevent any links on the pages from being followed.

In your use case, the meta tag you provided should work perfectly.

Scorpion Ghost said:

The webpage has this tag:

Code:

<meta name="robots" content="noindex,follow"/>

The robots.txt has this in it:

HTML:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

When I run the webpage in Google inspection tool, I get this:

View attachment 217746

We have "Indexing allowed? Yes"

Why?

Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why? Why?

Click to expand...

You have indexing allowed. Robots.txt overrides individual metatags for pages.

Your Robots.txt says:
disallow: /wp-admin
Which only disallows indexing the admin page.

That is enough to override the metatag for your page.

Don't rely on meta directives. They are only taken as "crawl hints". If you want a page not to be indexed have a single source of truth in your robots.txt. that's the first place any law abiding crawler visits and listens to.

BlogPro said:

You are correct.

Adding nofollow to a meta tag will prevent any links on the pages from being followed.

In your use case, the meta tag you provided should work perfectly.

Click to expand...

But it doesn't. And it's been almost 6 months and I'm getting pissed off.

This is a long-ish thread - https://www.blackhatworld.com/seo/google-indexing-pages-and-categories-should-i-noindex.1387286/

But the gist of it is I've set the "noindex" meta tag on various pages (tag pages, tag pages pagination, homepage pagination, category pages pagination), and yet they're all still indexed. The ones that are not indexed are just because Google didn't find them (or maybe wouldn't index them anyway).

Ashk881 said:

You have indexing allowed. Robots.txt overrides individual metatags for pages.

Your Robots.txt says:
disallow: /wp-admin
Which only disallows indexing the admin page.

That is enough to override the metatag for your page.

Click to expand...

According to what I read, that is not the case. I can remove the robots.txt file completely, which allows all indexing, but Google should respect the meta tag on individual pages.

I've had the noindex meta tag for almost 6 months on some pages, and I didn't have a Robots.txt file AT ALL, and the pages are still indexed.

I only created a robots.txt file yesterday in hopes that maybe google now takes notice of the "noinxed" on the pages and deindexes them. I even added one page to the google removal tool, to try and get at least one removed, to see if it works.

Were these pages indexed before you added the tag?

BlogPro said:

Were these pages indexed before you added the tag?

Click to expand...

Yes.

But I did the inspecting thing last night. You can see the date in my screenshot. It's supposed to crawl the page as it is now.

So unless Google is showing old info from the page, then google considers the page to be "Indexing allowed? Yes"

But... why?

URL inspection, by default, does not conduct a live test - it runs a test on the page as in Google's indexed. You have to click "Test Live URL". But seeing the recent time-stamp in the screenshot, I am assuming you've already done.

This leaves only one thing - there is something wrong with how the tags are being added on your page.

1. Have you cleared the cache? Is Googlebot being served the latest iteration of the page?
2. Are their conflicting tags in your header? (maybe added by a plugin or something).

To test this theory, I added the noindex tag to one of my previously indexed - non-performing pages in the SERPs and ran a test on the Live URL. This is my result, almost instantly.

BlogPro said:

URL inspection, by default, does not conduct a live test - it runs a test on the page as in Google's indexed. You have to click "Test Live URL". But seeing the recent time-stamp in the screenshot, I am assuming you've already done.

This leaves only one thing - there is something wrong with how the tags are being added on your page.

1. Have you cleared the cache? Is Googlebot being served the latest iteration of the page?
2. Are their conflicting tags in your header? (maybe added by a plugin or something).

To test this theory, I added the noindex tag to one of my previously indexed - non-performing pages in the SERPs and ran a test on the Live URL. This is my result, almost instantly.

Click to expand...

I've cleared the cache many times since I started trying to noindex these pages some 6 months ago.

But I just deleted cache again, then did inspect URL, and then did a Live Test. Page is still indexable.

Here is the tag in page source:



But like my "Contact us" page, which has the "noindex" meta tag added in the same way, that one is not indexed. And google says:

Indexing allowed? No: 'noindex' detected in 'robots' meta tag

Guess we'll have to do it like last time.

You add me to GSC, plus show me the site. I fix things for you.

We're both happier.

BlogPro said:

Guess we'll have to do it like last time.

You add me to GSC, plus show me the site. I fix things for you.

We're both happier.

Click to expand...

Oh jesus

PM

give us feedback what was wrong

Fixed for OP.

The culprit here was the Jetpack mobile theme.

Any custom filters he added to his site weren't being carried forward to the mobile version.

Google scraped using the Mobile bot and hence wasn't being served the customizations and hence kept including it in its index.

Disabled Jetpack mobile theme, everything works as expected.

BlogPro said:

Fixed for OP.

The culprit here was the Jetpack mobile theme.

Any custom filters he added to his site weren't being carried forward to the mobile version.

Google scraped using the Mobile bot and hence wasn't being served the customizations and hence kept including it in its index.

Disabled Jetpack mobile theme, everything works as expected.

Click to expand...

Indeed. He did his magic once again and fixed it. Thank youuuuuuuuuuuuuuu