Googlebot Still Crawling Old 301 URLs—Best Way to Stop It?

scenari0

scenari0

Registered Member
Joined
Jan 16, 2020
Messages
68
Reaction score
7
A few years ago, I 301-redirected thousands of URLs from https://example.com/photo/unique-label to consolidate content (https://example.com/product/unique-label). Even after 4 years, Googlebot is still crawling the old /photo/ URLs frequently.

I’ve checked in Ahrefs, and these old URLs aren’t receiving backlinks. I’m considering a few options to stop the crawls:
  1. Change the 301 redirects to 410 (Gone permanently)
  2. Change the 301s to 410, wait 1-2 years, and then add the URLs to robots.txt
  3. Add the URLs to robots.txt directly
Which option would be most effective, or is there a better alternative?
 
Advertise on BHW
Are you sure you don't have internal links to your old URLs?
 
praetserge said:
Are you sure you don't have internal links to your old URLs?
Click to expand...
Good question. I've been checking it for months. When inspecting the old URLs in GSC, I get as referrer:
- https://example.com/product/unique-label (the new consolidated HTTPS URL, but it does not content the link to the old URL into the HTML code)
- http://example.com/photo/unique-label (the old no-HTTPS version, which now 301-redirects to the HTTPS version)
- https://example.com/photo/unique-label (the old HTTPS version, which now 301-redirects to the HTTPS version)

Additionally, when I inspect the old URLS, the canonical (according to GSC) is the new consolidated HTTPS URL.

And the old URLs are not in the sitemap.
 
I use LinkWhisper and it tells me what internal pages link to each of my pages so I'd change those links to the new URLs.
If some of your articles link to your old URLs then google will go to the old URL then to the new one.

OK, I just checked my site as I have some 301 redirects... Recent redirect (3-5 days ago), my old URL is NOT indexed in google anymore, only the new one and I have no internal or external links to the old URL.
 
Change the 301 redirects to 410 for certain URLs to indicate to Google that these pages are permanently gone, which should lower their crawl frequency over time. If old URLs are still crawled after 1-2 years, consider using robots.txt to block them. Additionally, verify your internal links with tools like LinkWhisper or Screaming Frog to remove any hidden links to the old URLs.
 
praetserge said:
I use LinkWhisper and it tells me what internal pages link to each of my pages so I'd change those links to the new URLs.
If some of your articles link to your old URLs then google will go to the old URL then to the new one.

OK, I just checked my site as I have some 301 redirects... Recent redirect (3-5 days ago), my old URL is NOT indexed in google anymore, only the new one and I have no internal or external links to the old URL.
Click to expand...
Perhaps the issue that I consolidated a big website, with both HTTP and HTTPs versions
- http://example.com/photo/label --> https://example.com/product/label (300,000 URLs)
- https://example.com/photo/label --> https://example.com/product/label (300,000 URLs)
- http://amp.example.com/photo/label --> https://example.com/product/label (300,000 URLs)
- https://amp.example.com/photo/label --> https://example.com/product/label (300,000 URLs)

Some of the redirects were 2x301, and Google may be still crawling them.

Your experience (if you redirect any URL, Google must not index it anymore) makes a lot of sense.
 
Ding Dong said:
Change the 301 redirects to 410 for certain URLs to indicate to Google that these pages are permanently gone, which should lower their crawl frequency over time. If old URLs are still crawled after 1-2 years, consider using robots.txt to block them. Additionally, verify your internal links with tools like LinkWhisper or Screaming Frog to remove any hidden links to the old URLs.
Click to expand...
I will try with LinkWhishper or Screaming Frog, thank you. But GSC should tell me about any hidden link, right?
 
scenari0 said:
Perhaps the issue that I consolidated a big website, with both HTTP and HTTPs versions
- http://example.com/photo/label --> https://example.com/product/label (300,000 URLs)
- https://example.com/photo/label --> https://example.com/product/label (300,000 URLs)
- http://amp.example.com/photo/label --> https://example.com/product/label (300,000 URLs)
- https://amp.example.com/photo/label --> https://example.com/product/label (300,000 URLs)

Some of the redirects were 2x301, and Google may be still crawling them.

Your experience (if you redirect any URL, Google must not index it anymore) makes a lot of sense.
Click to expand...
As I said before in my previous post, I actually checked my own website and I don't have old pages indexed anymore... only the new ones.

Just like you I have many redirects... http to https, then I changed site URLs, I also moved and renamed some posts, and as I can see none of the old URLs are indexed in GSC.
 
Run a crawl with Screaming Frog and head over Visualizations to see what URL are linking to that one
 
I've been doing further research:
- by using Screaming Frog
- by using 'find' in the shell script of the server to find any file containing links to '/photo/'
- by using in Google: inurl:photo site:example.com

No current webpage in my server is links to '/photo/'. However, all the old URLs indexed by Google are linked (according to GSC) by webpages that were crawled... in 2020!

So Google is still crawling the links seen in its cache of 2020. I understand that I must not maintain such 301 redirects, and it's ok if I set 410 for them.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top