Google indexing pages and categories, should I noindex?

[Scorpion Ghost]

Okay, so a few replies above I shared what code I added in functions.php to add the "noindex" tag to those pagination pages.

Unfortunately for some reason I couldn't get it to add the tag with double quotes. Just stupid code shit really.

So I followed this other tutorial and added the noindex with the double quotes and everything - https://www.quickonlinetips.com/archives/2020/04/noindex-archive-subpages-yoast-seo-plugin/

In fact, this method adds the noindex tag under the Yoast section of meta tags. So it's like added functionality to Yoast or something? Well, added functionality for a function that Yoast removed in 2018 I believe. Stupid stupid stupid.

Anyway, I'll do a quick check of all my pages to make sure the noindex tag is added where I want it, and is not present where I don't want it. Then I forget about it and check again in a few weeks to see if google finally deindexed these pagination pages taking up and stealing space on the search engine from my good articles and posts

#SEOStillSucks

 

[Scorpion Ghost]

My last attempt didn't work, these pages are still indexed. It's crazy.

 

[Scorpion Ghost]

Turns out my website doesn't have a robots.txt file at all.

From my reading, this should not be a problem. It just means Google will try to index everything, but it will respect any "noindex" meta tag on individual pages.

But it's not working. I think the only reason some of my pages are not indexed is because Google never found them because I never linked to them anywhere. But whichever pages it did find, it indexed, completely ignoring the "noindex" meta tag.

I think this because I ran a few of the pages that are not indexed on Google inspection tool and I got:

URL is not on Google
This page is not in the index, but not because of an error. See the details below to learn why it wasn't indexed.

But I also read someone mentioned that without a robots.txt file Google may behave weird. Whatever, it's all retarded with seo anyway.

So I created a robots.txt file:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

But I think this won't work either, because when I run in the inspection tool one of the pages that do have a "noindex" meta tag, I get this:

Crawl
Last crawl May 18, 2022, 5:05:52 PM
Crawled as Googlebot smartphone
Crawl allowed? Yes
Page fetch Successful
Indexing allowed? Yes

HOW? HOW THE FUCK IS INDEXING ALLOWED? WHAT ABOUT MY "NOINDEX" METATAG?

See this is why I like spamming, because things make sense when you spam. This seo is a total jerkoff...

Anyway, I submitted a request to REMOVE PAGE for this page:

https://mysite.com/page/7/
Let's see if that does anything.

 

[Scorpion Ghost]

Well well well, this is mostly likely fixed. I just need to wait for Google to crawl these pages again, and to make sure they remove them from the serps.

The problem was the Jetpack mobile module. It was showing google the mobile version of my site, where there was no "noindex" metatag on these pages, so google was not deindexing them because it wasn't getting to see the "noindex" tag at all.

Who knew eh

So @BlogPro knew, after a few minutes, and fixed the problem. A big round of applause please, and thanks again

 

[Scorpion Ghost]

@BlogPro so here we are 10 days later, and Google is still indexing those pages.

How long do you think it will be until they remove them from the SERPS?

 

[BlogPro]

@BlogPro so here we are 10 days later, and Google is still indexing those pages.

How long do you think it will be until they remove them from the SERPS?

Give it time. Crawl budget is a reality.

Check the "Last crawled" date on in GSC.

If you're in a rush, use the temporary removal tool to get rid of those pages. And given the noindex tag, they won't be reindexed.

 

[Scorpion Ghost]

Give it time. Crawl budget is a reality.

Check the "Last crawled" date on in GSC.

If you're in a rush, use the temporary removal tool to get rid of those pages. And given the noindex tag, they won't be reindexed.

Yeah, I'm not in a rush. But I would like for this to finally get concluded, so that I can remove 2 entries from my browser's bookmarks

I checked one of these pages:

Last crawl
May 18, 2022, 5:05:52 PM

So actually I did a huge template change on the website in the past few days, also edited some posts a little bit, so maybe google takes notice that the site is "alive and kicking" and does some good crawling soon.

Can't be bothered with the removal tool. There's like, I don't know, about 50-100 pages probably that need to go. But if google doesn't deindex these pages at some point I'll collect them all and do the removal tool thing, to get it over with.

Cheers

 

[Cutoff]

You can use yoast seo if use wordpress for example for have noindex in the /tag pages and use Google search console for remove from the serp the /tag pages of you site. It work and fast

 

[Scorpion Ghost]

You can use yoast seo if use wordpress for example for have noindex in the /tag pages and use Google search console for remove from the serp the /tag pages of you site. It work and fast

It works, but Yoast removed the functionality to add noindex to navigation pages.

homepage.com//page/7/
homepage.com/category/page/3/
homepage.com/category/subcategory/page/5/

You can't do this with Yoast, they removed the functionality a few years ago, according to some articles and forum discussions I found online.

To do it, you need to use a code in your functions.php file, which can also be found online. And this is what I used, and it works.

My problem was the Jetpack mobile theme not allowing google's robots to see the noindex tag, I believe because they were being served the mobile version of the site, and it didn't have the noindex tags that I added.

But it's all been resolved now, I think, and now I'm just waiting on google to crawl all these pages and remove them from the SERPS. Of course, my site not being very popular, and google sucking, I may end up having to get all of the pages I want removed from the SERPS and use the removal tool or whatever to nudge google to remove the pages.

Still waiting...

 

[Cutoff]

I use yoast and i can add the noindex to /tag, /pages, /author, /category. After the noindex, you have send to Google search console the new sitemap? For example, i have in the sitemap only post, pages, category

 

[Seo Captain]

There is no loss even if they are indexing category.

 

[Scorpion Ghost]

There is no loss even if they are indexing category.

If you do a google search for my site like "site:mysite.com" the ENTIRE first page is filled with homepage navigation results (site.com/page/2/, site.com/page/3/, etc).

I don't like it.

 

[Black Belt Spodermen]

I just did a site:website.com search on Google for one of my Wordpress sites, and I found the first 2 pages of google populated by Pages and Categories and Sub-categories.

For example:

https://website.com/page/7/https://website.com/page/12/
https://website.com/whatever/https://website.com/stuff/
https://website.com/whatever/other/https://website.com/whatever/stuff/
My actual posts start from the end of page 2 and onwards. But even on page 3 and page 4 there are some page/category/subcategory pages indexed.

This looks kinda shitty, I don't know.

Should I add a noindex on all these pages? What do you guys do? What's the 2022 best advice?

I had the same problem with my site previously. I added the no index line to my robots.txt that disallows the tags and category pages.

 

[Scorpion Ghost]

I had the same problem with my site previously. I added the no index line to my robots.txt that disallows the tags and category pages.

Actually I do want the category pages indeed, but not the category navigation pages.

site.com/category/mycategory - YES

site.com/category/mycategory/page/2/ - NO

I feel is a nice balance.

 

[Black Belt Spodermen]

Actually I do want the category pages indeed, but not the category navigation pages.

site.com/category/mycategory - YES

site.com/category/mycategory/page/2/ - NO

I feel is a nice balance.

I don't know if it will work but do you think disallowing the url- site.com/category/mycategory/page/ will work?

 

[Scorpion Ghost]

I don't know if it will work but do you think disallowing the url- site.com/category/mycategory/page/ will work?

I added a noindex tag to all those pages that I want removed from the serps. I'm just waiting for google to crawl them (and hoping they pick up on the tag and actually deindex them).

 

[Seo Captain]

If you do a google search for my site like "site:mysite.com" the ENTIRE first page is filled with homepage navigation results (site.com/page/2/, site.com/page/3/, etc).

I don't like it.

If you do not like it, even other visitors searching the site on google will not like it.

 

[Scorpion Ghost]

@BlogPro

So I google my site to check if google deindexed those pages.

I see the pages are still indexed, for example - site.com/page/7/

I check the page in Webmasters, it says:

URL is not on Google
This page is not in the index, but not because of an error. See the details below to learn why it wasn't indexed.

I check details:

Last crawl Aug 12, 2022, 4:06:11 PM
Crawled as Googlebot smartphone
Crawl allowed? Yes
Page fetch Successful
Indexing allowed? No: 'noindex' detected in 'robots' meta tag

But the page is still in the serps.

Then I thought maybe the problem is that I'm googling while logged in with that Google account. So maybe it's showing it to me, but not to the world. So I open Chrome where I'm logged into a different Google account, and the page is still there. I try Brave and Opera, the page is still there. Then I thought maybe the IP address, so I use a US proxy with a Firefox browser and not logged into Google, and the page is still fucking there

It's one of three things:

1) Google just crawled the page yesterday and saw the 'noindex' tag, and it needs a few days to deindex the page
2) The "noindex" tag will tell a search engine not to index a page, but it won't make it remove an already indexed page from the serps
3) Google is just fucking with me

So what is it? What now?


---

EDIT

I'm thinking it's 1, because for example this one is not in the serps - site.com/page/6/

Everything the same, only:

Last crawl Aug 4, 2022, 2:59:04 PM

And this one is not there - site.com/page/3

Last crawl Aug 11, 2022, 9:48:07 PM

But this one IS THERE - site.com/page/5

Last crawl Jul 25, 2022, 11:03:14 AM

And this one IS there - site.com/page/10

Last crawl Jul 25, 2022, 5:54:54 PM

And this one IS there - site.com/page/15 (yeah no shit, this page doesn't exist on the site, leads to PAGE NOT FOUND)

Last crawl May 20, 2022, 3:00:26 AM

---

Okay now I'll wait to see about this one - site.com/page/7/

Google says it crawled it yesterday. It says it's not in the serps, but it is! Okay, I understand, you need a few hours to remove it from the serps, because technology has not advanced to the point where you can do it instantly after BEING FUCKING TOLD TO!!!

And I'll use the removal tool for this one because, the page doesn't exist, your crawlers can't get to a page that doesn't exist, you idiot google - site.com/page/15

So we wait. We'll see what happens...

 

[Brandon Burling]

You can keep them,its not a problem

 

[Scorpion Ghost]

You can keep them,its not a problem

Well I beg to differ.

I don't wanna keep them. I want... exactly what I want.

=

Index these pages, don't index these other pages.