1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to block all bots/crawlers apart from search engines?

Discussion in 'Black Hat SEO' started by WeWontBeAlone, Jun 21, 2017.

  1. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    472
    Likes Received:
    39
    How to block all bots/crawlers apart from search engines?

    Need it with .htaccess
     
  2. Onmef

    Onmef Newbie

    Joined:
    Jun 19, 2017
    Messages:
    31
    Likes Received:
    0
    Gender:
    Male
    https://github.com/search?q=bot+blocker+htaccess
     
  3. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    472
    Likes Received:
    39
  4. KHer0

    KHer0 Supreme Member

    Joined:
    Mar 22, 2011
    Messages:
    1,230
    Likes Received:
    1,160
    Occupation:
    Architect
    It's more complicated than one line in htaccess. You need PHP. Cuz probably the bots you want the block are using Googlebot agent name when visiting your websites.

    So, you want to check first if it's really google (W/E search engine you are targeting) then serve the content you want. If not, then block the content.
     
  5. redarrow

    redarrow Elite Member

    Joined:
    Apr 1, 2013
    Messages:
    4,307
    Likes Received:
    984
  6. yellowcat

    yellowcat Regular Member

    Joined:
    Aug 27, 2015
    Messages:
    299
    Likes Received:
    164
    Location:
    internet 24/7
    Home Page:
    You don't xD
     
  7. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    472
    Likes Received:
    39
  8. laur.laurix

    laur.laurix Power Member

    Joined:
    May 8, 2013
    Messages:
    592
    Likes Received:
    222
    Occupation:
    Reverse Engineering Maniac
    Location:
    Mars
    U clearly a noob...so try all in one seo plugin and block it from there.
     
  9. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    472
    Likes Received:
    39
    Not what I asked lol
     
  10. laur.laurix

    laur.laurix Power Member

    Joined:
    May 8, 2013
    Messages:
    592
    Likes Received:
    222
    Occupation:
    Reverse Engineering Maniac
    Location:
    Mars
    it`s exactly what u asked. All in one seo has an bad bot blocker feature.

     
  11. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    472
    Likes Received:
    39
    I asked for .htaccess