1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Internet Archive now ignores Robotos.txt for all web sites

Discussion in 'BlackHat Lounge' started by RightFootFanatic, Apr 24, 2017.

  1. RightFootFanatic

    RightFootFanatic Regular Member

    Joined:
    May 31, 2015
    Messages:
    347
    Likes Received:
    192
    Occupation:
    DevOps
    Location:
    Whimsyshire
    Internet Archive announced last week that they will stop referring to robots.txt files for all web sites

    see: https://blog.archive.org/2017/04/17...arch-engines-dont-work-well-for-web-archives/