403, 404, 410 How Fast De-Index?
- Thread starter Joe 10
- Start date
Google Offers Advice on 404 and 410 Status Codes
https://www.searchenginejournal.com/google-404-status/254429/
https://www.searchenginejournal.com/google-404-status/254429/
I know they do.
But I am asking for your personal experiences.
I read dozens of articles from google. My stuff has not been deindexed in a while now, so I dont trust what I read from google anymore. I want to know what you guys personal experience of what worked.
I don't think in a black hat forum people should listen to what google says. They do what works.
But I am asking for your personal experiences.
I read dozens of articles from google. My stuff has not been deindexed in a while now, so I dont trust what I read from google anymore. I want to know what you guys personal experience of what worked.
I don't think in a black hat forum people should listen to what google says. They do what works.
Unreliable Witness
Senior Member
- Apr 21, 2016
- 837
- 627
Google first needs to crawl the URLs to detect that they no longer exist. So you should go through the same process as you would if you were indexing them.
However, Google also doesn't deindex URLs on the first crawl, in case you've a made a mistake in giving the URL a 4XX code. So you need to make sure you get it to crawl the URLs a second time a couple of weeks after the first crawl.
That's personal experience - if you follow that process, its possible in two weeks. The key is to get Google to crawl the URLs regularly.
However, Google also doesn't deindex URLs on the first crawl, in case you've a made a mistake in giving the URL a 4XX code. So you need to make sure you get it to crawl the URLs a second time a couple of weeks after the first crawl.
That's personal experience - if you follow that process, its possible in two weeks. The key is to get Google to crawl the URLs regularly.
Google indexed 100+ of my directories such as /wp-upload/2018/ etc.
I set 403 to them 5 weeks ago, they are still indexed.
I submitted URL removal request 5 weeks ago, they are hidden from search results but still indexed (I can see it in GSC).
So you are saying I should change 403 to 404?
I set 403 to them 5 weeks ago, they are still indexed.
I submitted URL removal request 5 weeks ago, they are hidden from search results but still indexed (I can see it in GSC).
So you are saying I should change 403 to 404?
If I block them in robots google will not crawl them so it cant deindex them.
I had set them to 403 for 5 weeks and nothing changed, still indexed.
Now2 minutes ago I changed it to 404. I hope it's the best way.
Google will attempt to crawl them, seeing that the robot is blocked they will remove it from its index.
Yes, I do, some time ago I forgot to block google from indexing my wp plugin directory and all kinds of shit showed up in google index (CSS, scripts and whatnot), so I blocked it in robots file and everything was gone. Don't remember though how long it took. But the point is that after an attempt to crawl it again and seeing that it's blocked they will remove everything from index what was indexed before. Google will make an attempt to crawl it because they've done it in the past. To speed things up you need to find a way to force google bot to visit that directory again, on its own it may take some time. Either in GSC or using links.
most people using wordpress have this robots.txt:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
this doesn't seem to me it's blocking /wp-plugin/ do you have this blocked?
Anyway the directories indexed in my case where all the folders with images, /uploads/2017/02 etc. If I block this in robots, this may affect my image files.
Now I set it to 404, I hope it's right and it will get that stuff deindexed.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
this doesn't seem to me it's blocking /wp-plugin/ do you have this blocked?
Anyway the directories indexed in my case where all the folders with images, /uploads/2017/02 etc. If I block this in robots, this may affect my image files.
Now I set it to 404, I hope it's right and it will get that stuff deindexed.
Among other things this is what's in my robot file:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
This basically tells all robots (be it google, bing, yahoo, and whatnot) not to crawl those directories. If they can't crawl it, they can't index the content what's in there. And if the content has been indexed before, they'll remove it as they are no longer allowed to crawl it.
Blocking robots from /uploads/2017/02 will affect your image files but if you're concerned about that why would you want to deindex and making them 404 or 410?
If you have in /uploads/2017/02 only selective things that you don't want to be indexed you have to go on URL by URL bases. Alternatively, you can use the 'deindex function' in GSC. This doesn't really deindex anything, it just hides selected URL from appearing in google index (SERP). This should help you to speed things up until google takes note of the 404/410. But with 100+ URLs you have your work cut out for you if doing it manually. Another solution is to move the things that you don't want to index from /uploads/2017/02 to a separate folder and block robots from crawling it. However, if they are images that appear on your posts you'll have to update the URLs and it may affect your SEO.
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
This basically tells all robots (be it google, bing, yahoo, and whatnot) not to crawl those directories. If they can't crawl it, they can't index the content what's in there. And if the content has been indexed before, they'll remove it as they are no longer allowed to crawl it.
Blocking robots from /uploads/2017/02 will affect your image files but if you're concerned about that why would you want to deindex and making them 404 or 410?
If you have in /uploads/2017/02 only selective things that you don't want to be indexed you have to go on URL by URL bases. Alternatively, you can use the 'deindex function' in GSC. This doesn't really deindex anything, it just hides selected URL from appearing in google index (SERP). This should help you to speed things up until google takes note of the 404/410. But with 100+ URLs you have your work cut out for you if doing it manually. Another solution is to move the things that you don't want to index from /uploads/2017/02 to a separate folder and block robots from crawling it. However, if they are images that appear on your posts you'll have to update the URLs and it may affect your SEO.
You guys dont' understand.
Of course I have images in these folders. And of course they should be crawled.
The problem is that in Google Search Results if you search:
site:yoursite.com
you find indexed these URLS
yourwebsite.com/uploads/2017/02
yourwebsite.com/uploads/2015/04
and 100s like this.
This is not URLs you want to appear in Search Results.
They are thin content.
I changed the status from 403 to 404 now, I re-submitted the sitemap.
Actually I set the status to 410 in .htaccess, but when I check with https status it gives a 404.
If you guys have more suggestions, I would appreciate it.
I never had an issue with google indexing my
yourwebsite.com/uploads/
or
yourwebsite.com/uploads/2017/02
There must be something wrong with how your WP is set up.
Here are a couple of links, not sure if it'll help but could point you in the direction:
https://support.google.com/webmasters/thread/16974145?hl=en
https://webmasters.stackexchange.co...wp-content-directories-and-all-internal-files
yourwebsite.com/uploads/
or
yourwebsite.com/uploads/2017/02
There must be something wrong with how your WP is set up.
Here are a couple of links, not sure if it'll help but could point you in the direction:
https://support.google.com/webmasters/thread/16974145?hl=en
https://webmasters.stackexchange.co...wp-content-directories-and-all-internal-files