How do you guys go about blocking SEO bots?
I'm talking to those who have private SEO/blog networks...
I mean, it makes sense but I just can't figure out how to successfully do it!
There's 2 approaches, and I've read a bunch of information on BHW and other forums
1) Block via robots.txt
2) Block via htaccess
1) is risky in my opinion. Anyone can see the robots.txt if they type it in, and I've read that bots also ignore it
2) is the better approach, and it's what I've been trying to do, however it is NOT working me either.
I'm finding that Majestic and Ahrefs are still picking up the domains.
Here's one of the codes I've tried:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^rogerbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^exabot [OR]
RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [OR]
RewriteCond %{HTTP_USER_AGENT} ^dotbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^gigabot [OR]
RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot
RewriteRule .* - [F]
^^ Doesn't work. Does anyone with experience know what is wrong with it?
And...does anyone have a htaccess code that works?
If you have something that is definitely working but don't want to share it on here, PM me as I'd be willing to pay.
Though this Forum is all about sharing, so if you have something that is working for you right now...share away. I'm sure there's a few others in my position!
Thanks
I'm talking to those who have private SEO/blog networks...
I mean, it makes sense but I just can't figure out how to successfully do it!
There's 2 approaches, and I've read a bunch of information on BHW and other forums
1) Block via robots.txt
2) Block via htaccess
1) is risky in my opinion. Anyone can see the robots.txt if they type it in, and I've read that bots also ignore it
2) is the better approach, and it's what I've been trying to do, however it is NOT working me either.
I'm finding that Majestic and Ahrefs are still picking up the domains.
Here's one of the codes I've tried:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^rogerbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^exabot [OR]
RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [OR]
RewriteCond %{HTTP_USER_AGENT} ^dotbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^gigabot [OR]
RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot
RewriteRule .* - [F]
^^ Doesn't work. Does anyone with experience know what is wrong with it?
And...does anyone have a htaccess code that works?
If you have something that is definitely working but don't want to share it on here, PM me as I'd be willing to pay.
Though this Forum is all about sharing, so if you have something that is working for you right now...share away. I'm sure there's a few others in my position!
Thanks