1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Purchased domain wont let me go wayback due to robots.txt HELP!

Discussion in 'Domain Names & Parking' started by 23cromij, Sep 16, 2014.

  1. 23cromij

    23cromij Junior Member

    Joined:
    Sep 7, 2013
    Messages:
    101
    Likes Received:
    7
    Occupation:
    Entrepreneur
    Location:
    UK
    Hi guys hopefully someone can help.

    i just purchased a domain and when i try to use wayback to see what the site used to look like it says the information is hidden due to robots.txt

    i have not yet connected the nameserver to my host will this help?
     
  2. arganrecords

    arganrecords Elite Member

    Joined:
    Oct 12, 2013
    Messages:
    1,670
    Likes Received:
    2,067
    Occupation:
    Go Hard Or Go Home
    Location:
    Italy
    Have you checked the waybackmachine before buy the domain?
    If the site was blocked by the robots.txt means that the owner did not want to index the site or make it access by spiders.

    The site is indexed?
     
  3. 23cromij

    23cromij Junior Member

    Joined:
    Sep 7, 2013
    Messages:
    101
    Likes Received:
    7
    Occupation:
    Entrepreneur
    Location:
    UK
    no i checked after,

    i understand that they used robots.txt to block access but i dont know why

    i believe the site is not indexed?
     
  4. 23cromij

    23cromij Junior Member

    Joined:
    Sep 7, 2013
    Messages:
    101
    Likes Received:
    7
    Occupation:
    Entrepreneur
    Location:
    UK
    how can i remove the robots.txt so i can view what it looked like on waybackmachine
     
  5. bignose

    bignose Jr. VIP Jr. VIP

    Joined:
    Dec 6, 2009
    Messages:
    708
    Likes Received:
    109
    Gender:
    Male
    It's stupid that you didn't check the archive before purchase.

    Anyway, now it may be because of two reasons: 1. The old owner asked archive.org to hide the content; 2. Your registrar parking does this. For example Gandi always does this. In this case, in order to see the archive you just need to send your DNS to some hosting.
     
  6. 23cromij

    23cromij Junior Member

    Joined:
    Sep 7, 2013
    Messages:
    101
    Likes Received:
    7
    Occupation:
    Entrepreneur
    Location:
    UK
    its not my registrar because iv just installed wordpress and directed my nameservers to my hosting?

    why should i have checked before?
     
  7. bignose

    bignose Jr. VIP Jr. VIP

    Joined:
    Dec 6, 2009
    Messages:
    708
    Likes Received:
    109
    Gender:
    Male
    Well, in this case you would know whether there is something in the archive or not, before purchasing the domain ))
     
  8. 23cromij

    23cromij Junior Member

    Joined:
    Sep 7, 2013
    Messages:
    101
    Likes Received:
    7
    Occupation:
    Entrepreneur
    Location:
    UK
    the site already had states on opensiteexplorer so i knew it had been used before
     
  9. hardcorebiker

    hardcorebiker Jr. VIP Jr. VIP

    Joined:
    Sep 6, 2011
    Messages:
    1,380
    Likes Received:
    1,071
    Occupation:
    Digital Marketer
    Location:
    Bangkok, Thailand
    Try using screenshots.com. Its not as good as archive.org but works sometimes.
     
  10. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Jr. VIP

    Joined:
    Nov 10, 2012
    Messages:
    12,582
    Likes Received:
    34,690
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page: