1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Internet Archive now ignores Robotos.txt for all web sites

Discussion in 'BlackHat Lounge' started by RightFootFanatic, Apr 24, 2017.

  1. RightFootFanatic

    RightFootFanatic Regular Member

    Joined:
    May 31, 2015
    Messages:
    319
    Likes Received:
    180
    Occupation:
    DevOps
    Location:
    Whimsyshire
    Internet Archive announced last week that they will stop referring to robots.txt files for all web sites

    see: https://blog.archive.org/2017/04/17...arch-engines-dont-work-well-for-web-archives/