[Question] How to tell Google to skip crawling certain URLs.

[superKruv]

Hey BHW.

I have 30,000 pages overall, about 10K of them are not relevant to be indexed.
I don't want Google to spend crawling time on those 10K pages, as I imagine it delays indexing the good URLs.

I have a list of the URLs. Is there a way to tell Google (Through console maybe) to skip crawling those 10K?

Thanks.

Edit: If using Robots.txt to achieve that, can I whitelist good URLs instead of blacklisting bad ones?

 

[Roger Marquez]

Edit: If using Robots.txt to achieve that, can I whitelist good URLs instead of blacklisting bad ones?

It is the best way but, just because you´re implementing a disallow directive in an URL it doesn´t mean you´re blacklisting it.

 

[superKruv]

It is the best way but, just because you´re implementing a disallow directive in an URL it doesn´t mean you´re blacklisting it.

Thanks for your answer, I wonder if there is a way to tell him what to crawl rather than telling him what not to. I thought about this solution, will it work?

User-agent: Googlebot
Allow: [list of urls I want to whitelist]
Disallow: /

 

[Adeebkhatri]

You can prevent a page or other resource from appearing in Google Search by including a noindex meta tag or header in the HTTP response. When Googlebot next crawls that page and sees the tag or header, Google will drop that page entirely from Google Search results, regardless of whether other sites link to it.

 

[sageshark]

If you want to prevent certain pages from being crawled then how about adding this tag in the header?

<meta name="googlebot" content="noindex">

Edit : There are ways to bulk edit multiple posts meta tags via .CSV.

 

[Roger Marquez]

If you want to prevent certain pages from being crawled then how about adding this tag in the header?

<meta name="googlebot" content="noindex">

This doesn´t prevent googlebot from not crawling that URL.

You might tell Googlebot NOT to index that page but it will still crawl it.

 

[superKruv]

If you want to prevent certain pages from being crawled then how about adding this tag in the header?

<meta name="googlebot" content="noindex">

Edit : There are ways to bulk edit multiple posts meta tags via .CSV.

I don't mind whether they are indexed or not. Instead, I care about the process power Google crawl bot invest in my domain. I want to utilize it to my good urls rather than urls that will end up in "crawled not indexed".

 

[sageshark]

I want to utilize it to my good urls rather than urls that will end up in "crawled not indexed".

I understand your point. You should start looking for some good indexing services, they might help you with this.

In my case I just focus on 3 things -

1) I do proper manual interlinking between the articles those are crawled but not indexed.
2) I submit them to social media regularly.
3) Resubmit the pages for indexation (GSC allows 10 pages max per day. Pls don't laugh, but I do that. Its slow but it works. I believe that it is better to do it than leave it undone.)

I do this hassle because I do not want to go for any indexation service available out there. But if you are ready to test them please go ahead and share your experience with us.