1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

If you have Robot.txt issue with wayback archive then just sign in then it will be gone

Discussion in 'Black Hat SEO' started by Peter Ngo, Apr 4, 2015.

  1. Peter Ngo

    Peter Ngo Jr. VIP Jr. VIP

    Joined:
    Apr 23, 2013
    Messages:
    2,145
    Likes Received:
    1,723
    Occupation:
    I browse BHW for a living
    Location:
    The Internet
    For the past 1-2 weeks (after their recently update), i have had countless "Page cannot be crawled or displayed due to robots.txt." error and it was sooooooo damn annoying.

    You have to refresh a few times to see the content. The solution i found is staying signed in and the error will be gone :| damn should have known this sooner.

    Am i the only one that encountered this issue?
     
    • Thanks Thanks x 2
  2. Lares

    Lares Junior Member

    Joined:
    Jun 27, 2011
    Messages:
    190
    Likes Received:
    74
    No you are not the only one. I am getting this too since yesterday. Will try to create an account. Thanks for the tip. It will save me lots of nerves :)
     
    • Thanks Thanks x 1
  3. cottonwolf

    cottonwolf Regular Member

    Joined:
    Jan 20, 2015
    Messages:
    469
    Likes Received:
    239
    oh, got lots of errors due to this error text when buying some domains this week
     
    • Thanks Thanks x 1
  4. Peter Ngo

    Peter Ngo Jr. VIP Jr. VIP

    Joined:
    Apr 23, 2013
    Messages:
    2,145
    Likes Received:
    1,723
    Occupation:
    I browse BHW for a living
    Location:
    The Internet
    Well, i just did it for like few hundreds time and the error came back :-ss shiittt

    Logging in doesn't help much.

    Anyone got solution for this?
     
  5. IPaidForThisName

    IPaidForThisName Regular Member

    Joined:
    Feb 15, 2013
    Messages:
    386
    Likes Received:
    72
    Location:
    UK
    Thank you, I thought something out was out of order.
     
  6. coccodimamma

    coccodimamma Newbie

    Joined:
    Aug 29, 2013
    Messages:
    24
    Likes Received:
    9
    Hi,
    same here. I registered but I get the same error.
     
  7. Zipix

    Zipix Registered Member

    Joined:
    Apr 16, 2014
    Messages:
    86
    Likes Received:
    43
    I want to join the list with people having archive.org issues (the robot.txt one). No idea if there is a solution to the problem or we will have to wait until the staff of the Wayback Machine comes up with a fix, if at all..
     
  8. SupremeBoner

    SupremeBoner Junior Member

    Joined:
    Jan 8, 2015
    Messages:
    114
    Likes Received:
    20
    Yes.

    Seems ever since they switched to their mobile version of the site the whole database has been fuckey.

    I have refreshed the page 80fcking times and still it doesnt work.

    this is why i dont liek change
     
  9. T2tkid

    T2tkid Elite Member

    Joined:
    Oct 13, 2010
    Messages:
    3,820
    Likes Received:
    1,742
    I have been trying this shit for days now and it is not working! Keep reloading donkey times! It is so annoying!! I hope whatever it is, they will fix it soon.

    OP, i registered, keep refreshing, but the f....kin pages will not show..
     
  10. francis1017

    francis1017 Supreme Member

    Joined:
    Feb 26, 2013
    Messages:
    1,492
    Likes Received:
    346
    Yeah me too. I keep getting this, so I am using screenshots.com.
     
  11. randybishop

    randybishop Elite Member

    Joined:
    Aug 23, 2013
    Messages:
    1,921
    Likes Received:
    795
    Occupation:
    I was born a poor black child
    Location:
    Mississippi
    Thank you!!!!!!!!!!!!!!!
     
  12. Peter Ngo

    Peter Ngo Jr. VIP Jr. VIP

    Joined:
    Apr 23, 2013
    Messages:
    2,145
    Likes Received:
    1,723
    Occupation:
    I browse BHW for a living
    Location:
    The Internet
    I tried a few tests today and things seem to be normal now. :-?

    Pray to god please this error don't come back.

    Oh it seems screenshot.com is pretty nice as well. might switch over to that if they have broader database.
     
  13. IPaidForThisName

    IPaidForThisName Regular Member

    Joined:
    Feb 15, 2013
    Messages:
    386
    Likes Received:
    72
    Location:
    UK
    Now it keeps logging me out 24/7
     
  14. francis1017

    francis1017 Supreme Member

    Joined:
    Feb 26, 2013
    Messages:
    1,492
    Likes Received:
    346
    still getting problems with this..
     
  15. SupremeBoner

    SupremeBoner Junior Member

    Joined:
    Jan 8, 2015
    Messages:
    114
    Likes Received:
    20
    signing in does nothing to help the problem

    also, screenshots.com is inferior
     
  16. salo123

    salo123 Junior Member

    Joined:
    Jan 1, 2014
    Messages:
    168
    Likes Received:
    51
    Location:
    Place in the Sun!
    I also have the issue, and no announcement from wayback. Really annoying..
     
  17. Lares

    Lares Junior Member

    Joined:
    Jun 27, 2011
    Messages:
    190
    Likes Received:
    74
    It seems you get this error only with domains that are not registered anywhere. All the other domains work fine.
     
  18. accelerator_dd

    accelerator_dd Jr. VIP Jr. VIP

    Joined:
    May 14, 2010
    Messages:
    2,448
    Likes Received:
    1,010
    Occupation:
    SEO
    Location:
    IM Wonderland
    I've been getting the same for the past 3-4 days, had to edit my bots to keep trying until they get a page - gets tricky because some domains are really locked by robots and no matter how many times you try it will keep saying the same :p.

    I am sure they'll fix it pretty soon.
     
    • Thanks Thanks x 1
  19. SupremeBoner

    SupremeBoner Junior Member

    Joined:
    Jan 8, 2015
    Messages:
    114
    Likes Received:
    20
    Dis be google.

    cutts slipped archive a fiver to stop em from displaying old domains to thwart pbn
     
  20. TBSdomains

    TBSdomains BANNED BANNED

    Joined:
    Jun 18, 2014
    Messages:
    63
    Likes Received:
    3
    this is stupid, one of the most important uses of archive.org is to access webpages which are not online anymore, I hope this is a temporarily bug.