Linus persson
Registered Member
- Jan 15, 2017
- 68
- 22
<html>
<head>
<title>...</title>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>
- Jun 20, 2013
- 11,503
- 8,911
He ask for ht access code not this one. OP check this it may help https://www.blackhatworld.com/seo/a...s-and-all-others-from-crawling-a-site.584470/
Linus persson
Registered Member
- Jan 15, 2017
- 68
- 22
I know, but that is usually an easy way to do it.
Google Prince
Regular Member
- Dec 24, 2015
- 209
- 182
Here ya go buddy 
Code:
SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
SetEnvIfNoCase User-Agent .*exabot.* bad_bot
SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>
issorc
Senior Member
- Feb 20, 2016
- 826
- 765
He wants to block crawlers like Ahrefs and Majestic and they don't give a fuck whether you'd like your site to be indexed or not.
Werombi
Registered Member
- May 26, 2015
- 64
- 33
I know this post few months old though would like to know whats the difference with the quoted htaccess and the below?
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^USER_AGENT [NC,OR]
RewriteRule ^.* - [F,L]
I use wordpress and was given a list of agents to block wwith the above conditions. If 1 better over another?
Bishal Tripathi
Newbie
- Oct 26, 2016
- 26
- 2
How to block "ubersuggest" crawler.?