1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Purchased domain wont let me go wayback due to robots.txt HELP!

Discussion in 'Domain Names & Parking' started by 23cromij, Sep 16, 2014.

  1. 23cromij

    23cromij Registered Member

    Joined:
    Sep 7, 2013
    Messages:
    95
    Likes Received:
    7
    Occupation:
    Entrepreneur
    Location:
    UK
    Hi guys hopefully someone can help.

    i just purchased a domain and when i try to use wayback to see what the site used to look like it says the information is hidden due to robots.txt

    i have not yet connected the nameserver to my host will this help?
     
  2. arganrecords

    arganrecords Elite Member

    Joined:
    Oct 12, 2013
    Messages:
    1,611
    Likes Received:
    1,904
    Occupation:
    I think I'm Marketer
    Location:
    Italy
    Have you checked the waybackmachine before buy the domain?
    If the site was blocked by the robots.txt means that the owner did not want to index the site or make it access by spiders.

    The site is indexed?
     
  3. 23cromij

    23cromij Registered Member

    Joined:
    Sep 7, 2013
    Messages:
    95
    Likes Received:
    7
    Occupation:
    Entrepreneur
    Location:
    UK
    no i checked after,

    i understand that they used robots.txt to block access but i dont know why

    i believe the site is not indexed?
     
  4. 23cromij

    23cromij Registered Member

    Joined:
    Sep 7, 2013
    Messages:
    95
    Likes Received:
    7
    Occupation:
    Entrepreneur
    Location:
    UK
    how can i remove the robots.txt so i can view what it looked like on waybackmachine
     
  5. bignose

    bignose Power Member

    Joined:
    Dec 6, 2009
    Messages:
    599
    Likes Received:
    90
    Gender:
    Male
    It's stupid that you didn't check the archive before purchase.

    Anyway, now it may be because of two reasons: 1. The old owner asked archive.org to hide the content; 2. Your registrar parking does this. For example Gandi always does this. In this case, in order to see the archive you just need to send your DNS to some hosting.
     
  6. 23cromij

    23cromij Registered Member

    Joined:
    Sep 7, 2013
    Messages:
    95
    Likes Received:
    7
    Occupation:
    Entrepreneur
    Location:
    UK
    its not my registrar because iv just installed wordpress and directed my nameservers to my hosting?

    why should i have checked before?
     
  7. bignose

    bignose Power Member

    Joined:
    Dec 6, 2009
    Messages:
    599
    Likes Received:
    90
    Gender:
    Male
    Well, in this case you would know whether there is something in the archive or not, before purchasing the domain ))
     
  8. 23cromij

    23cromij Registered Member

    Joined:
    Sep 7, 2013
    Messages:
    95
    Likes Received:
    7
    Occupation:
    Entrepreneur
    Location:
    UK
    the site already had states on opensiteexplorer so i knew it had been used before
     
  9. hardcorebiker

    hardcorebiker Jr. VIP Jr. VIP

    Joined:
    Sep 6, 2011
    Messages:
    1,250
    Likes Received:
    1,045
    Occupation:
    Digital Marketer
    Location:
    Bangkok, Thailand
    Try using screenshots.com. Its not as good as archive.org but works sometimes.
     
  10. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,124
    Likes Received:
    28,563
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page: