1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to block all bots/crawlers apart from search engines?

Discussion in 'Black Hat SEO' started by WeWontBeAlone, Jun 21, 2017.

  1. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    489
    Likes Received:
    39
    How to block all bots/crawlers apart from search engines?

    Need it with .htaccess
     
  2. Onmef

    Onmef Newbie

    Joined:
    Jun 19, 2017
    Messages:
    33
    Likes Received:
    0
    Gender:
    Male
    https://github.com/search?q=bot+blocker+htaccess
     
  3. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    489
    Likes Received:
    39
  4. KHer0

    KHer0 Supreme Member

    Joined:
    Mar 22, 2011
    Messages:
    1,343
    Likes Received:
    1,226
    Occupation:
    Architect
    It's more complicated than one line in htaccess. You need PHP. Cuz probably the bots you want the block are using Googlebot agent name when visiting your websites.

    So, you want to check first if it's really google (W/E search engine you are targeting) then serve the content you want. If not, then block the content.
     
  5. redarrow

    redarrow Elite Member

    Joined:
    Apr 1, 2013
    Messages:
    5,207
    Likes Received:
    1,190
  6. yellowcat

    yellowcat Regular Member

    Joined:
    Aug 27, 2015
    Messages:
    304
    Likes Received:
    174
    Location:
    internet 24/7
    Home Page:
    You don't xD
     
  7. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    489
    Likes Received:
    39
  8. laur.laurix

    laur.laurix Power Member

    Joined:
    May 8, 2013
    Messages:
    703
    Likes Received:
    262
    Occupation:
    Reverse Engineering Maniac
    Location:
    Mars
    U clearly a noob...so try all in one seo plugin and block it from there.
     
  9. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    489
    Likes Received:
    39
    Not what I asked lol
     
  10. laur.laurix

    laur.laurix Power Member

    Joined:
    May 8, 2013
    Messages:
    703
    Likes Received:
    262
    Occupation:
    Reverse Engineering Maniac
    Location:
    Mars
    it`s exactly what u asked. All in one seo has an bad bot blocker feature.

     
  11. WeWontBeAlone

    WeWontBeAlone Regular Member

    Joined:
    Jan 13, 2016
    Messages:
    489
    Likes Received:
    39
    I asked for .htaccess