need to block ahrefs

nonai · Feb 19, 2014

using htaccess, I want to block as many backliink checking tools as possible. I have already done some research on this (including searching this forum) but I have not been able to find a solution. I have found several proposed solutions, but not one that's confirmed working by more than one person.

I will post them here:

solution 1:

Code:

BrowserMatchNoCase rogerbot bad_bot
BrowserMatchNoCase exabot bad_bot
BrowserMatchNoCase MJ12bot bad_bot
BrowserMatchNoCase dotbot bad_bot
BrowserMatchNoCase gigabot bad_bot
BrowserMatchNoCase AhrefsBot bad_bot
Order Deny,Allow
Deny from env=bad_bot

solution 2:

Code:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^rogerbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^exabot [OR]
RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [OR]
RewriteCond %{HTTP_USER_AGENT} ^dotbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^gigabot [OR]
RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot
RewriteRule .* - [F]

solution 3:

Code:

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ^rogerbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^exabot [OR]
RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [OR]
RewriteCond %{HTTP_USER_AGENT} ^dotbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^gigabot [OR]
RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot
RewriteRule ^.* - [F,L]

solution 4:

Code:

SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
SetEnvIfNoCase User-Agent .*exabot.* bad_bot
SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>

In addition to these, I want to block
BLEXbot
Blekkobot
SEOkicks-Robot

I dont know anything about apache and these are codes that I just found online. I have no clue which one would work and which one would not. If you could just let me know which solution is best (and maybe your reasoning), that would be really appreciated.

BHW-News · Feb 19, 2014

Found these with quick search:

https://ahrefs.com/robot/index.php - Used before.

http://www.blackhatworld.com/blackh...majestic-ahrefs-all-others-crawling-site.html

innozemec · Feb 19, 2014

search on google for their IPs and simply block those ips in the htaccess and thats it

JonnyBorneo · Feb 19, 2014

robots.txt

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

This will block everything, except googlebot

lover · Feb 19, 2014

Plugins like spyderspanker can do this. Can block ahrefs + many more.

innozemec · Feb 19, 2014

JonnyBorneo said:
robots.txt

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

This will block everything, except googlebot

that is not blocking anything, it is just telling the bot you don't want it to continue, however it may not obey it if it wants..

BlueSteele · Feb 20, 2014

Ahrefs is annoying because it goes by different names. "ahrefsbot" and "sitebot" for example. I believe there are more too, but I don't know them. Here is the list that I block:

AhrefsBot
Alexibot
Aqua_Products
asterias
b2w/0.1
BackDoorBot/1.0
BecomeBot
BlekkoBot
BlowFish/1.0
Bookmark search tool
BotALot
BuiltBotTough
Bullseye/1.0
BunnySlippers
CheeseBot
CherryPicker
Copernic
CopyRightCheck
cosmos
Crescent
Crescent Internet ToolPak HTTP OLE Control v.1.0
DittoSpyder
dotbot
dumbot
EmailCollector
EmailSiphon
EmailWolf
Enterprise_Search
EroCrawler
es
exabot
ExtractorPro
FairAd Client
Flaming AttackBot
Foobot
Gaisbot
GetRight/4.2
gigabot
grub
grub-client
Harvest/1.5
Hatena Antenna
hloader
Pubcon | Pubcon Search, Social Media, Affiliate Marketing Conferences bot
WebmasterWorld News and Discussion for the Web Professional bot
httplib
humanlinks
ia_archiver
InfoNaviRobot
Iron33/1.0.2
JennyBot
Jetbot
Kenjin Spider
Keyword Density/0.9
larbin
LexiBot
libWeb/clsHTTP
LinkextractorPro
LinkScan/8.1a Unix
LinkWalker
LNSpiderguy
looksmart
lwp-trivial
lwp-trivial/1.34
Mata Hari
Microsoft URL Control
MIIxpc
Mister PiX
MJ12bot
moget
MSIECrawler
naver
NetAnts
NetMechanic
NICErsPRO
Nutch
Offline Explorer
Openbot
Openfind
Openfind data gathere
Oracle Ultra Search
PerMan
ProPowerBot/2.14
ProWebWalker
psbot
Python-urllib
QueryN Metasearch
Radiation Retriever 1.1
RepoMonkey
RepoMonkey Bait & Tackle/v1.01
RMA
rogerbot
scooter
searchpreview
Sitebot
SiteSnagger
sootle
SpankBot
spanner
Stanford
Stanford Comp Sci
Stanford CompClub
Stanford CompSciClub
Stanford Spiderboys
suzuran
Szukacz/1.4
Teleport
TeleportPro
Telesoft
Teoma
The Intraformant
TheNomad
toCrawl/UrlDispatcher
True_Robot
turingos
URL Control
URL_Spider_Pro
URLy Warning
VCI
VCI WebViewer VCI WebViewer Win32
Web Image Collector
WebAuto
WebBandit
WebCopier
WebEnhancer
WebmasterWorld Extractor
WebmasterWorldForumBot
WebSauger
Website Quester
Webster Pro
WebStripper
WebVac
WebZip
Wget
WWW-Collector-E
Xenu
Xenu's
Zeus

Here is a pretty sweet plugin that will tell you what bots are visiting your site: http://wordpress.org/plugins/seo-crawlytics/

DamageX · Feb 20, 2014

I blocked ahrefs through 301 by useragent, didn't work. I have since found a better option, which also blocks by IP.

http://www.blackhatworld.com/blackh...ng-through-301-a-post6440563.html#post6440563

elner · Feb 20, 2014

a noob qustion here. why did you guy want to block these ? You don't want competitor knows your site details ?

just4ivaylo · Feb 20, 2014

Yes, you don't want to have a competitor checking out your backlink profile and so on. You can "steal" links that way and get a link on the same website that they're on. It works pretty well sometimes.

BlueSteele · Feb 20, 2014

Could you block ahrefs from seeing only a part of your link profile...for example, just my social signals, press releases or haha guest posts. Fool them into thinking that is what is pushing me, instead of all the HPR backlinks. Just to fuck with them.

DamageX · Feb 20, 2014

BlueSteele said:
Could you block ahrefs from seeing only a part of your link profile...for example, just my social signals, press releases or haha guest posts. Fool them into thinking that is what is pushing me, instead of all the HPR backlinks. Just to fuck with them.

You can only block it on sites you own.

BlueSteele · Feb 20, 2014

Would very much be my own site. Any thoughts?

DamageX said:
You can only block it on sites you own.

DamageX · Feb 20, 2014

BlueSteele said:
Would very much be my own site. Any thoughts?

Here you go, you're welcome.

DamageX said:
I blocked ahrefs through 301 by useragent, didn't work. I have since found a better option, which also blocks by IP.

http://www.blackhatworld.com/blackh...ng-through-301-a-post6440563.html#post6440563

bkdigitalsoft · Feb 20, 2014

Why you want to block the site through ahref ?

ChokeStone · Feb 20, 2014

Blocking bots doesn't block people from seeing your backlinks. It only blocks these bots from crawling sites that YOU own. That means the only links that you can potentially hide from your competitors are links from your OWN private network. everything else is being seen.

rumah · Dec 23, 2017

hello is there anybody can help me to block ahrefs from being able to crawl my website?

MisterF · Dec 23, 2017

rumah said:
hello is there anybody can help me to block ahrefs from being able to crawl my website?

Have you tried all the methods mentioned above ?

SEOMOOD · Dec 23, 2017

To hide PBN or 301 redirects before ahrefs, the best way to do it is cloaking

rumah · Dec 26, 2017

i do like this in my robot.txt prntscr[dot]com/hsbi40 , hello freind can you help me have i done this correctly?

need to block ahrefs

BANNED

Super Moderator

Elite Member

Regular Member

Elite Member

Elite Member

Regular Member

Elite Member

BANNED

Regular Member

Regular Member

Elite Member

Regular Member

Elite Member

Senior Member

Registered Member

Newbie

Repeat Selling Out MP - Doxxing - Harass Comps.

Regular Member

Newbie

Main Menu

Marketplace

Making Money

BlackHat World