1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Disallow all crawlers except Google

Discussion in 'Black Hat SEO' started by xpofkex, Jun 10, 2014.

  1. xpofkex

    xpofkex Junior Member

    Joined:
    Aug 18, 2010
    Messages:
    158
    Likes Received:
    35
    Hi,

    How to disallow all crawlers (Ahrefs,Yahoobot,Bingbot,MajesticSEObot and etc..) except Google with .htaccess? I know how to do with robots.txt but I need with .htaccess.

    Thank you!
     
  2. irdeto

    irdeto Regular Member

    Joined:
    Mar 18, 2010
    Messages:
    390
    Likes Received:
    163
    +1 here for this request.
    I haven't done much research on it and would also like to know.
     
  3. Final Infinity

    Final Infinity Junior Member

    Joined:
    Mar 29, 2014
    Messages:
    146
    Likes Received:
    73
    I am not sure how to do it with .htaccess but if you are using wp you can use a plugin called Spyder Spanker. Costs a little bit of money but well worth it imo. You can choose any kind of bots you want to let access your website and those that you don't. Hope that helps at least a little bit.
     
  4. tony_d

    tony_d Elite Member

    Joined:
    Jun 22, 2013
    Messages:
    2,581
    Likes Received:
    3,163
    Location:
    1600 Amphitheatre Parkway, Mountain View CA
    You did specify htaccess, but a robots.txt file with the following will do the trick nicely

    Code:
    User-agent: Googlebot
    Disallow:
    
    User-agent: msnbot
    Disallow:
    
    User-agent: Slurp
    Disallow:
    
    User-agent: Teoma
    Disallow:
    
    User-agent: rogerbot
    Disallow: /
    
    User-agent: exabot
    Disallow: /
    
    User-agent: MJ12bot
    Disallow: /
    
    User-agent: dotbot
    Disallow: /
    
    User-agent: gigabot
    Disallow: /
    
    User-agent: AhrefsBot
    Disallow: /
    
    User-agent: *
    Disallow: /
    
    
     
    • Thanks Thanks x 2
  5. xpofkex

    xpofkex Junior Member

    Joined:
    Aug 18, 2010
    Messages:
    158
    Likes Received:
    35
    Yes, I know how to block with robots.txt but I need block with .htaccess (not all bots checks robots.txt) I need to block them from my all website.
     
  6. the_demon

    the_demon Jr. Executive VIP

    Joined:
    Nov 23, 2008
    Messages:
    3,177
    Likes Received:
    1,563
    Occupation:
    Search Engine Marketing
    Location:
    The Internet
  7. xpofkex

    xpofkex Junior Member

    Joined:
    Aug 18, 2010
    Messages:
    158
    Likes Received:
    35
  8. cashcorp

    cashcorp Regular Member

    Joined:
    Feb 8, 2008
    Messages:
    430
    Likes Received:
    270
    Home Page:
    Use the .htaccess, robots.txt is useless for Majestic/Ahrefs etc.

    Further, even "Botblocking" via htaccess fails. You'll want to drop in the relevant IP ranges (When you can find them) as well.

     
    • Thanks Thanks x 1
  9. domainmadness

    domainmadness Senior Member

    Joined:
    Jun 22, 2011
    Messages:
    1,036
    Likes Received:
    334
    You can use this, just redirect them to google.

    RewriteEngine On
    RewriteCond %{REQUEST_URI} !/robots.txt$
    RewriteCond %{HTTP_USER_AGENT} ^.*BLEXBot.*$ [NC,OR]
    ..... more bots .....
    RewriteRule ^.*.* http://google.com/ [L]

    I hope this helps.
     
  10. xpofkex

    xpofkex Junior Member

    Joined:
    Aug 18, 2010
    Messages:
    158
    Likes Received:
    35
    Thanks! I will try this soon.
     
  11. TropicalSun

    TropicalSun Regular Member

    Joined:
    Apr 17, 2011
    Messages:
    258
    Likes Received:
    34
    Occupation:
    Perpetual innovator
    I've also had a problem blocking access to ahrefs/majestic, but especially ahrefs.

    Any idea why Ahrefs is being so difficult?