1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How To Block SEO Bots From Finding My Network?

Discussion in 'Black Hat SEO' started by MDirect, Mar 20, 2014.

  1. MDirect

    MDirect Junior Member

    Joined:
    Feb 6, 2012
    Messages:
    102
    Likes Received:
    29
    How do you guys go about blocking SEO bots?

    I'm talking to those who have private SEO/blog networks...

    I mean, it makes sense but I just can't figure out how to successfully do it!

    There's 2 approaches, and I've read a bunch of information on BHW and other forums

    1) Block via robots.txt
    2) Block via htaccess

    1) is risky in my opinion. Anyone can see the robots.txt if they type it in, and I've read that bots also ignore it

    2) is the better approach, and it's what I've been trying to do, however it is NOT working me either.

    I'm finding that Majestic and Ahrefs are still picking up the domains.

    Here's one of the codes I've tried:

    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} ^rogerbot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^exabot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^dotbot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^gigabot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot


    RewriteRule .* - [F]

    ^^ Doesn't work. Does anyone with experience know what is wrong with it?

    And...does anyone have a htaccess code that works?

    If you have something that is definitely working but don't want to share it on here, PM me as I'd be willing to pay.

    Though this Forum is all about sharing, so if you have something that is working for you right now...share away. I'm sure there's a few others in my position!

    Thanks
     
  2. xha44a

    xha44a Power Member

    Joined:
    Dec 2, 2012
    Messages:
    532
    Likes Received:
    444
    Hi

    Something I've been looking at (for spam bots) is to ip ban via htaccess for not honouring the robots file. On my cell so can't paste code but works like this

    1 robots says nobody goes to /admin
    2 has hidden link to /admin on top of homepage
    3 bad bot simply can't resist, visits /admin
    4 /admin is a php script that adds the visiting ip to a blocked by htaccess
    5 result. Bad bot which doesn't honour robots gets his a** banned from your site pronto.

    Not exactly what you are after but may help with bots.
     
    • Thanks Thanks x 1
  3. kbklash

    kbklash Senior Member

    Joined:
    Jun 2, 2012
    Messages:
    1,140
    Likes Received:
    449
    Location:
    in my BMW to my BHW
    Home Page:
    blocking via robot.txt is easy IMO.
    User-agent: googlebotDisallow: /
    etc will do the job.

    dotbot
    gigabot
    exabot all these are associated with seo related bots.
     
  4. innosoft

    innosoft Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 25, 2008
    Messages:
    1,633
    Likes Received:
    639
    Occupation:
    Software Developer, SEO
    Location:
    Office
    Home Page:
    there is lots of tutorial to how to block a bot.. y dont u use google :)
     
  5. vonSteinberg

    vonSteinberg Registered Member

    Joined:
    Feb 24, 2014
    Messages:
    92
    Likes Received:
    7
    I am also interested in the same thing, please post here if you test something and it works.
     
  6. MDirect

    MDirect Junior Member

    Joined:
    Feb 6, 2012
    Messages:
    102
    Likes Received:
    29
    Still not found something that works for me - though I'm following up on a couple of ideas

    Anyone else on here who can help?