A Robots.txt error

lordmortimer

Newbie

May 17, 2017

5

0

• May 22, 2017

• #1

I've made a mistake while creating my "robots.txt" file and wrote the Disallow command, so I'm now unable to see my website's archived content. I immediately corrected the file but I still see no change in the Archive: how much do I have to wait and, more importantly, is the archived content forever gone now?

 

vexxatu

Newbie

Jan 22, 2016

48

9

• May 22, 2017

• #2

You mean you disallowed your /archive/ directory? Usually even if you dissalow your entire site, google will pick it up again pretty quickly. If it's a new site, it will take some days, probably max a week. If it's an established site, it can Google can index it again as low as 24hr (tested).

 

lordmortimer

Newbie

May 17, 2017

5

0

• May 22, 2017
• Thread Starter Thread Starter

• #3

I disallowed the entire site and, when I noticed something was wrong, immediately changed back. Problem is that if I try to access the Internet Archive now it says that I cannot view snapshots because of robots.txt

I guess that by tomorrow my site will be crawled again but some 10-year-old content was only available via IA and I'm not sure if I will be able to watch it again now.

 

Gogol

Administrator

Staff member

Moderator

Executive VIP

Jr. VIP

Sep 10, 2010

11,805

26,657

• May 22, 2017

• #4

May be going to webmaster, recrawling and indexing the robots.txt could help..

In fact, i dont think anything will help if google bot does not decide to visit you anytime sooner.. the best thing to do would be to just wait and see what happens..


Similar thread

Code:

https://moz.com/community/q/recovering-from-robots-txt-error

 

Last edited: May 22, 2017

djp371

Elite Member

Apr 20, 2009

2,086

355

• May 24, 2017

• #5

Try using "Fetch as Google' in your Webmaster Tools. By this, you'll be asking Google to fetch your updated robots.txt file quickly.

 

lordmortimer

Newbie

May 17, 2017

5

0

• May 26, 2017
• Thread Starter Thread Starter

• #6

Everything is ok: I just changed robots.txt and then posted a link on facebook and twitter. Content on the Internet Archive is available once again too. Thank you.

 

lordmortimer

Newbie

May 17, 2017

5

0

• Jun 30, 2017
• Thread Starter Thread Starter

• #7

Everything WAS ok. Just recently the archived website is unavailable again. And this time I didn't touch the robots.txt file. Triple checked: it allows everything.