I need to block the robots in .htaccess so that I don't have to use a plugin like spider spanker on the PBN domains. I want to block: majestic, ahrefs, opensite explorer, semrush, semalt as the main ones.
I've checked other sources and I found this:
htaccess
SetEnvIfNoCase User-Agent .rogerbot. dangerbot
SetEnvIfNoCase User-Agent .mj12bot. dangerbot
SetEnvIfNoCase User-Agent .ahrefsbot. dangerbot
SetEnvIfNoCase User-Agent .semrushbot. dangerbot
Order Allow,Deny
Allow from all
Deny from env=dangerbot
Robots.txt
User-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: rogerbot
Disallow:/
User-agent: SemrushBot
Disallow: /
Can anyone tell me on how to do this exactly and how can I test spoof as a different user agent and robots to confirm it blocks robots? Thank you, I appreciate any thoughts!
I've checked other sources and I found this:
htaccess
SetEnvIfNoCase User-Agent .rogerbot. dangerbot
SetEnvIfNoCase User-Agent .mj12bot. dangerbot
SetEnvIfNoCase User-Agent .ahrefsbot. dangerbot
SetEnvIfNoCase User-Agent .semrushbot. dangerbot
Order Allow,Deny
Allow from all
Deny from env=dangerbot
Robots.txt
User-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: rogerbot
Disallow:/
User-agent: SemrushBot
Disallow: /
Can anyone tell me on how to do this exactly and how can I test spoof as a different user agent and robots to confirm it blocks robots? Thank you, I appreciate any thoughts!