1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

A Robots.txt error

Discussion in 'Black Hat SEO' started by lordmortimer, May 22, 2017.

  1. lordmortimer

    lordmortimer Newbie

    Joined:
    May 17, 2017
    Messages:
    5
    Likes Received:
    0
    Gender:
    Male
    I've made a mistake while creating my "robots.txt" file and wrote the Disallow command, so I'm now unable to see my website's archived content. I immediately corrected the file but I still see no change in the Archive: how much do I have to wait and, more importantly, is the archived content forever gone now?
     
  2. vexxatu

    vexxatu Newbie

    Joined:
    Jan 22, 2016
    Messages:
    45
    Likes Received:
    7
    Gender:
    Male
    You mean you disallowed your /archive/ directory? Usually even if you dissalow your entire site, google will pick it up again pretty quickly. If it's a new site, it will take some days, probably max a week. If it's an established site, it can Google can index it again as low as 24hr (tested).
     
  3. lordmortimer

    lordmortimer Newbie

    Joined:
    May 17, 2017
    Messages:
    5
    Likes Received:
    0
    Gender:
    Male
    I disallowed the entire site and, when I noticed something was wrong, immediately changed back. Problem is that if I try to access the Internet Archive now it says that I cannot view snapshots because of robots.txt

    I guess that by tomorrow my site will be crawled again but some 10-year-old content was only available via IA and I'm not sure if I will be able to watch it again now.
     
  4. Gogol

    Gogol Jr. VIP Jr. VIP

    Joined:
    Sep 10, 2010
    Messages:
    3,478
    Likes Received:
    3,108
    Gender:
    Male
    May be going to webmaster, recrawling and indexing the robots.txt could help..

    In fact, i dont think anything will help if google bot does not decide to visit you anytime sooner.. the best thing to do would be to just wait and see what happens..


    Similar thread
    Code:
    https://moz.com/community/q/recovering-from-robots-txt-error  
     
    Last edited: May 22, 2017
  5. djp371

    djp371 Elite Member

    Joined:
    Apr 20, 2009
    Messages:
    2,082
    Likes Received:
    350
    Occupation:
    Internet marketing
    Location:
    United States
    Home Page:
    Try using "Fetch as Google' in your Webmaster Tools. By this, you'll be asking Google to fetch your updated robots.txt file quickly.
     
  6. lordmortimer

    lordmortimer Newbie

    Joined:
    May 17, 2017
    Messages:
    5
    Likes Received:
    0
    Gender:
    Male
    Everything is ok: I just changed robots.txt and then posted a link on facebook and twitter. Content on the Internet Archive is available once again too. Thank you.
     
  7. lordmortimer

    lordmortimer Newbie

    Joined:
    May 17, 2017
    Messages:
    5
    Likes Received:
    0
    Gender:
    Male
    Everything WAS ok. Just recently the archived website is unavailable again. And this time I didn't touch the robots.txt file. Triple checked: it allows everything.