1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

List of Crawlers to Block

Discussion in 'Black Hat SEO' started by WeWontBeAlone, May 28, 2017.

  1. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    489
    Likes Received:
    39
    Alright, so I set up my first 5 PBN sites, I haven't blocked crawlers or anything yet, does anybody have a good list of bots to block in robots.txt and also for my .htaccess ? I keep finding posts from 2012, 2013, etc but they're all a little bit different and i'd like a more up to date list.
     
  2. laur.laurix

    laur.laurix Power Member

    Joined:
    May 8, 2013
    Messages:
    744
    Likes Received:
    281
    Occupation:
    Reverse Engineering Maniac
    Location:
    Mars
    install all in one seo and check block bad bots. also u can edit the plugin take the bad bot list from it and block them using htaccess.
     
  3. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    489
    Likes Received:
    39
    That will block ahrefs, majestic, etc? would rather have a definite list
     
  4. laur.laurix

    laur.laurix Power Member

    Joined:
    May 8, 2013
    Messages:
    744
    Likes Received:
    281
    Occupation:
    Reverse Engineering Maniac
    Location:
    Mars
    yes. u can extract the list from the plugin itself.

     
  5. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    489
    Likes Received:
    39
    Anybody got a list? I'd like to do it manually.
     
  6. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    489
    Likes Received:
    39
    bump
     
  7. curiouskt

    curiouskt Regular Member

    Joined:
    Apr 5, 2017
    Messages:
    235
    Likes Received:
    133
    lol, each to their own I guess ;)
     
  8. WebzSuite

    WebzSuite Jr. VIP Jr. VIP

    Joined:
    Oct 22, 2016
    Messages:
    211
    Likes Received:
    37
    Gender:
    Male
    Location:
    UK
    Home Page:
    Like above, everyone will prefer to block different bots.

    You can find some starting list in here:

    user-agents (dot) org