Internet Archive now ignores Robotos.txt for all web sites

Discussion in 'BlackHat Lounge' started by RightFootFanatic, Apr 24, 2017.

  1. RightFootFanatic

    RightFootFanatic Regular Member

    Joined:
    May 31, 2015
    Messages:
    440
    Likes Received:
    243
    Occupation:
    DevOps
    Location:
    Whimsyshire
    Internet Archive announced last week that they will stop referring to robots.txt files for all web sites

    see: https://blog.archive.org/2017/04/17...arch-engines-dont-work-well-for-web-archives/