How To Block SEO Bots From Finding My Network?

MDirect

Junior Member
Joined
Feb 6, 2012
Messages
104
Reaction score
29
How do you guys go about blocking SEO bots?

I'm talking to those who have private SEO/blog networks...

I mean, it makes sense but I just can't figure out how to successfully do it!

There's 2 approaches, and I've read a bunch of information on BHW and other forums

1) Block via robots.txt
2) Block via htaccess

1) is risky in my opinion. Anyone can see the robots.txt if they type it in, and I've read that bots also ignore it

2) is the better approach, and it's what I've been trying to do, however it is NOT working me either.

I'm finding that Majestic and Ahrefs are still picking up the domains.

Here's one of the codes I've tried:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^rogerbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^exabot [OR]
RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [OR]
RewriteCond %{HTTP_USER_AGENT} ^dotbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^gigabot [OR]
RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot


RewriteRule .* - [F]

^^ Doesn't work. Does anyone with experience know what is wrong with it?

And...does anyone have a htaccess code that works?

If you have something that is definitely working but don't want to share it on here, PM me as I'd be willing to pay.

Though this Forum is all about sharing, so if you have something that is working for you right now...share away. I'm sure there's a few others in my position!

Thanks
 
Hi

Something I've been looking at (for spam bots) is to ip ban via htaccess for not honouring the robots file. On my cell so can't paste code but works like this

1 robots says nobody goes to /admin
2 has hidden link to /admin on top of homepage
3 bad bot simply can't resist, visits /admin
4 /admin is a php script that adds the visiting ip to a blocked by htaccess
5 result. Bad bot which doesn't honour robots gets his a** banned from your site pronto.

Not exactly what you are after but may help with bots.
 
blocking via robot.txt is easy IMO.
User-agent: googlebotDisallow: /
etc will do the job.

dotbot
gigabot
exabot all these are associated with seo related bots.
 
there is lots of tutorial to how to block a bot.. y dont u use google :)
 
I am also interested in the same thing, please post here if you test something and it works.
 
Still not found something that works for me - though I'm following up on a couple of ideas

Anyone else on here who can help?
 
Back
Top