1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Bot crawlers blocked on PBN

Discussion in 'Black Hat SEO' started by patlang, Oct 12, 2016.

  1. patlang

    patlang Junior Member

    Joined:
    Sep 3, 2015
    Messages:
    177
    Likes Received:
    11
    Hello,
    I am seeking for a list of bot crawlers which are recommended to be blocked on your PBN.
    Anyone could share a script?

    Thanks
     
  2. DigitalCon

    DigitalCon Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 27, 2014
    Messages:
    516
    Likes Received:
    87
    Gender:
    Male
    Occupation:
    Internet Research
    Location:
    Home
    Home Page:
    Its been shared on BHW quite a few times. You could search for it rather than creating a thread.
    Anyways, here:

    Code:
    SetEnvIfNoCase User-Agent .rogerbot. bad_bot
    SetEnvIfNoCase User-Agent .exabot. bad_bot
    SetEnvIfNoCase User-Agent .mj12bot. bad_bot
    SetEnvIfNoCase User-Agent .dotbot. bad_bot
    SetEnvIfNoCase User-Agent .gigabot. bad_bot
    SetEnvIfNoCase User-Agent .ahrefsbot. bad_bot
    SetEnvIfNoCase User-Agent .sitebot. bad_bot
    SetEnvIfNoCase User-Agent .semrushbot. bad_bot
    SetEnvIfNoCase User-Agent .ia_archiver. bad_bot
    SetEnvIfNoCase User-Agent .searchmetricsbot. bad_bot
    SetEnvIfNoCase User-Agent .seokicks-robot. bad_bot
    SetEnvIfNoCase User-Agent .sistrix. bad_bot
    SetEnvIfNoCase User-Agent .lipperhey spider. bad_bot
    SetEnvIfNoCase User-Agent .ncbot. bad_bot
    SetEnvIfNoCase User-Agent .backlinkcrawler. bad_bot
    SetEnvIfNoCase User-Agent .archive.org_bot. bad_bot
    SetEnvIfNoCase User-Agent .meanpathbot. bad_bot
    SetEnvIfNoCase User-Agent .pagesinventory. bad_bot
    SetEnvIfNoCase User-Agent .aboundexbot. bad_bot
    SetEnvIfNoCase User-Agent .spbot. bad_bot
    SetEnvIfNoCase User-Agent .linkdexbot. bad_bot
    SetEnvIfNoCase User-Agent .nutch. bad_bot
    SetEnvIfNoCase User-Agent .blexbot. bad_bot
    SetEnvIfNoCase User-Agent .ezooms. bad_bot
    SetEnvIfNoCase User-Agent .scoutjet. bad_bot
    SetEnvIfNoCase User-Agent .majestic-12. bad_bot
    SetEnvIfNoCase User-Agent .majestic-seo. bad_bot
    SetEnvIfNoCase User-Agent .dsearch. bad_bot
    SetEnvIfNoCase User-Agent .blekkobo. bad_bot
    SetEnvIfNoCase User-Agent .screaming frog seo spider/.* bad_bot
    SetEnvIfNoCase User-Agent .PHPCrawl. bad_bot
    SetEnvIfNoCase User-Agent .gocrawl. bad_bot
    SetEnvIfNoCase User-Agent .DigExt. bad_bot
    SetEnvIfNoCase User-Agent .DomainSONOCrawler. bad_bot
    SetEnvIfNoCase User-Agent .TweetmemeBot. bad_bot
    SetEnvIfNoCase User-Agent .OpenHoseBot/2.1. bad_bot
    SetEnvIfNoCase User-Agent .Kraken/0.1. bad_bot
    SetEnvIfNoCase User-Agent .-Java-. bad_bot
    SetEnvIfNoCase User-Agent .ubermetrics. bad_bot
    SetEnvIfNoCase User-Agent .best-seo. bad_bot
    SetEnvIfNoCase User-Agent .Synapse. bad_bot
    SetEnvIfNoCase User-Agent .Harvest. bad_bot
    SetEnvIfNoCase User-Agent .Harvester. bad_bot
    SetEnvIfNoCase User-Agent .harvester. bad_bot
    SetEnvIfNoCase User-Agent .harvest. bad_bot
    
    <Limit GET POST HEAD>
    
    Order Allow,Deny
    
    Allow from all
    
    Deny from env=bad_bot
    
    </Limit>
     
    • Thanks Thanks x 1
  3. samcram

    samcram Jr. VIP Jr. VIP

    Joined:
    Sep 10, 2014
    Messages:
    132
    Likes Received:
    35
    Occupation:
    SEO
    Location:
    Moscow
    Home Page:
    Here is big list - snipt.net/allthingsdem/very-long-list-to-block-bad-bots-in-htaccess-bc10af8f/
    I'm using this bot list in one project - works fine
     
    • Thanks Thanks x 1
  4. Pascal B

    Pascal B Newbie

    Joined:
    Oct 4, 2016
    Messages:
    18
    Likes Received:
    1
    Gender:
    Male
    Great post guys, i needed this as well
     
  5. patlang

    patlang Junior Member

    Joined:
    Sep 3, 2015
    Messages:
    177
    Likes Received:
    11
    Has anyone ever had trouble due to using this script on each and every PBN of yours?
     
  6. shieldsroadfirm

    shieldsroadfirm Newbie

    Joined:
    Oct 12, 2016
    Messages:
    8
    Likes Received:
    1
    Gender:
    Male
    Asked, shared and reshared multiple times already. There are many threads with that information as well as on the best ways of using them.
     
  7. patlang

    patlang Junior Member

    Joined:
    Sep 3, 2015
    Messages:
    177
    Likes Received:
    11
    Imagine I am on my mobile so would take me years to find a proper answer. Much more convenient was to create a new thread which apparently was not useful just for me
     
  8. TayaX

    TayaX Jr. VIP Jr. VIP

    Joined:
    Dec 13, 2010
    Messages:
    3,457
    Likes Received:
    1,931
    Occupation:
    Skype : TayaxBHW
    Location:
    France
    Home Page:
    Considering you'll need a computer to put this in the htaccess, I suppose you have a computer, then you can search.
     
    • Thanks Thanks x 2
  9. patlang

    patlang Junior Member

    Joined:
    Sep 3, 2015
    Messages:
    177
    Likes Received:
    11
    Thank you for pumping the post up with useless posts however I already got the answer.
    Farewell
     
  10. TayaX

    TayaX Jr. VIP Jr. VIP

    Joined:
    Dec 13, 2010
    Messages:
    3,457
    Likes Received:
    1,931
    Occupation:
    Skype : TayaxBHW
    Location:
    France
    Home Page:
    Yes please lecture me about posting for nothing... How ironic.