1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

List of Links crawlers bots

Discussion in 'Black Hat SEO' started by Johan bucco, Sep 15, 2016.

  1. Johan bucco

    Johan bucco Newbie

    Jul 19, 2013
    Likes Received:
    Hi there! I´m looking for a list of the links crawlers bots more commun, including sistrix, xovi, etc.,

    Does anybody have one? And which is the best way to block them in .htaccess? I´ve seen in some forums different ways to block them, which do you think is the best?