WNC-633901
Newbie
- Sep 18, 2017
- 3
- 0
It may be basic, but I redesigned my website and the old directory had to go, replaced by a set of new directories. I was having trouble getting Google to slurp down my pages and add them.
I put in a REMOVE request on the old directory, but maybe Googlebot kept wasting time trying to crawl the pages anyway, and finding 404 error pages. I suspect that REMOVE only stops Google from displaying the pages in search results, but still Google crawls them.
So I added a line in robots.txt to disallow the missing directory. It seems to be working, as Google is crawling more pages and indexing them too.
I put in a REMOVE request on the old directory, but maybe Googlebot kept wasting time trying to crawl the pages anyway, and finding 404 error pages. I suspect that REMOVE only stops Google from displaying the pages in search results, but still Google crawls them.
So I added a line in robots.txt to disallow the missing directory. It seems to be working, as Google is crawling more pages and indexing them too.