List of Crawlers to Block

Discussion in 'Black Hat SEO' started by WeWontBeAlone, May 28, 2017.

  1. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    489
    Likes Received:
    39
    Alright, so I set up my first 5 PBN sites, I haven't blocked crawlers or anything yet, does anybody have a good list of bots to block in robots.txt and also for my .htaccess ? I keep finding posts from 2012, 2013, etc but they're all a little bit different and i'd like a more up to date list.
     
  2. laur.laurix

    laur.laurix Senior Member

    Joined:
    May 8, 2013
    Messages:
    885
    Likes Received:
    362
    Occupation:
    Reverse Engineering Maniac
    Location:
    Mars
    install all in one seo and check block bad bots. also u can edit the plugin take the bad bot list from it and block them using htaccess.
     
  3. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    489
    Likes Received:
    39
    That will block ahrefs, majestic, etc? would rather have a definite list
     
  4. laur.laurix

    laur.laurix Senior Member

    Joined:
    May 8, 2013
    Messages:
    885
    Likes Received:
    362
    Occupation:
    Reverse Engineering Maniac
    Location:
    Mars
    yes. u can extract the list from the plugin itself.

     
  5. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    489
    Likes Received:
    39
    Anybody got a list? I'd like to do it manually.
     
  6. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    489
    Likes Received:
    39
    bump
     
  7. curiouskt

    curiouskt Regular Member

    Joined:
    Apr 5, 2017
    Messages:
    235
    Likes Received:
    133
    lol, each to their own I guess ;)
     
  8. WebzSuite

    WebzSuite Jr. VIP Jr. VIP

    Joined:
    Oct 22, 2016
    Messages:
    218
    Likes Received:
    38
    Gender:
    Male
    Location:
    UK
    Home Page:
    Like above, everyone will prefer to block different bots.

    You can find some starting list in here:

    user-agents (dot) org