1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Ahrefs, majestic etc. spider list for .htaccess

Discussion in 'Black Hat SEO' started by Neocortx, Sep 12, 2014.

  1. Neocortx

    Neocortx Regular Member

    Joined:
    Mar 2, 2013
    Messages:
    274
    Likes Received:
    173
    hey everybody,

    Some time ago I saw a thread where users shared a pretty big list for blocking spiders from most SEO bots in order to avoid competitors finding out about the PBN.

    Now that I need it, I just can't find it. I tried many different ways of searching, but nothing.

    Maybe someone has the link? Or any other place where I can find a big list.

    Thank you very much in advance.

    warm regards,
    neocortx
     
  2. Neocortx

    Neocortx Regular Member

    Joined:
    Mar 2, 2013
    Messages:
    274
    Likes Received:
    173
  3. coitza

    coitza Jr. VIP Jr. VIP Premium Member

    Joined:
    Oct 26, 2007
    Messages:
    2,565
    Likes Received:
    609
    Occupation:
    freelancer
    Home Page:
    If you're using wordpress as a platform i think ithemes security has all this stuff in it, saw that it filled my htaccess with pretty much similar entries.
     
    • Thanks Thanks x 1
  4. poliofighterzone

    poliofighterzone Newbie

    Joined:
    May 23, 2014
    Messages:
    18
    Likes Received:
    1
    Spyderspanker is I think what the plugin is called for wordpress that does it.
     
  5. SEO Power

    SEO Power Elite Member

    Joined:
    Jul 14, 2014
    Messages:
    2,637
    Likes Received:
    680
    Occupation:
    Self employed
    Location:
    Houston, TX
    You are right. There is a pro version of the plugin if you want more features, but the free plugin should be enough.