How to block robots in htaccess - NEED HELP!!

hangel

Newbie
Joined
Jun 17, 2015
Messages
16
Reaction score
3
I need to block the robots in .htaccess so that I don't have to use a plugin like spider spanker on the PBN domains. I want to block: majestic, ahrefs, opensite explorer, semrush, semalt as the main ones.

I've checked other sources and I found this:
htaccess

SetEnvIfNoCase User-Agent .rogerbot. dangerbot
SetEnvIfNoCase User-Agent .mj12bot. dangerbot
SetEnvIfNoCase User-Agent .ahrefsbot. dangerbot
SetEnvIfNoCase User-Agent .semrushbot. dangerbot

Order Allow,Deny
Allow from all
Deny from env=dangerbot




Robots.txt

User-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: rogerbot
Disallow:/
User-agent: SemrushBot
Disallow: /


Can anyone tell me on how to do this exactly and how can I test spoof as a different user agent and robots to confirm it blocks robots? Thank you, I appreciate any thoughts!
 
Awesome, thanks! With my next question, do you have any idea how I can test spoof as a different user agent and robots to confirm it blocks robots?
 
You can install addon for Firefox, "User Agent Switcher". Addon have a lot of standard user-agent strings, plus you can add custom strings manually.
 
Thanks TZ2011! I've installed the firefox add-on User Agent Switcher. I have also added custom strings manually as suggested. How do I exactly confirm that it has blocked those bots? Do I need to go to Majestic, Ahrefs, Semrush, etc to see if it still crawls my links? Thanks.
 
Last edited:
Some of these bastard link tools are using fake user agents. I had to start blocking by IP range too.
 
Yeah tools are now just randomizing their user-agent...I mean why wouldn't they if they want to be the most accurate tool?
 
set blocking of those bots in the .htaccess, you will see if they are blocked if you visit the site with custom user agents that are faking seo bots. if everything is allright, you should get 403 or 404 page, depend of your htaccess setup.
 
@hangel set blocking of those bots in the .htaccess, you will see if they are blocked if you visit the site with custom user agents that are faking seo bots. if everything is allright, you should get 403 or 404 page, depend of your htaccess setup.
 
Awesome! Thanks for your help TZ2011!
 
Back
Top