How to block ahrefs, semrush and others?

Rollfic

Rollfic

Elite Member
Joined
Dec 13, 2019
Messages
1,580
Reaction score
723
Hi, how do I block Ahrefs, semrush and other seo tools from crawling my site in order to avoid the obvious competitors?

At the same time I want to manually check traffic and other stats in Ahrefs, can it be done?
 
Advertise on BHW
Both Ahrefs and Semrush respect robots.txt entries. More details on their bots can be found at​
​
​
To block Ahrefs, your robots.txt should look like​
​
Rich (BB code): 
User-agent: AhrefsBot
Disallow: /
​
To block Semrush, your robots.txt should look like​
​
Rich (BB code): 
User-agent: SemrushBot
Disallow: /
​
Moz uses Rogerbot and Dotbot to crawl. Former for site analysis, the latter for Link index.​
​
Other services use their own custom bots.​
​
Here's a list to ban the most common crawlers -​
​
Code: 
User-agent: Rogerbot 
User-agent: Exabot 
User-agent: MJ12bot 
User-agent: Dotbot 
User-agent: Gigabot 
User-agent: AhrefsBot 
User-agent: BlackWidow
User-agent: ChinaClaw 
User-agent: Custo 
User-agent: DISCo 
User-agent: Download\ Demon 
User-agent: eCatch 
User-agent: EirGrabber 
User-agent: EmailSiphon 
User-agent: EmailWolf 
User-agent: Express\ WebPictures 
User-agent: ExtractorPro 
User-agent: EyeNetIE 
User-agent: FlashGet 
User-agent: GetRight 
User-agent: GetWeb! 
User-agent: Go!Zilla 
User-agent: Go-Ahead-Got-It 
User-agent: GrabNet 
User-agent: Grafula 
User-agent: HMView 
User-agent: HTTrack 
User-agent: Image\ Stripper 
User-agent: Image\ Sucker 
User-agent: Indy\ Library
User-agent: InterGET 
User-agent: Internet\ Ninja 
User-agent: JetCar 
User-agent: JOC\ Web\ Spider 
User-agent: larbin 
User-agent: LeechFTP 
User-agent: Mass\ Downloader 
User-agent: MIDown\ tool 
User-agent: Mister\ PiX 
User-agent: Navroad 
User-agent: NearSite 
User-agent: NetAnts 
User-agent: NetSpider 
User-agent: Net\ Vampire 
User-agent: NetZIP 
User-agent: Octopus 
User-agent: Offline\ Explorer 
User-agent: Offline\ Navigator 
User-agent: PageGrabber 
User-agent: Papa\ Foto 
User-agent: pavuk 
User-agent: pcBrowser 
User-agent: RealDownload 
User-agent: ReGet 
User-agent: SiteSnagger 
User-agent: SmartDownload 
User-agent: SuperBot 
User-agent: SuperHTTP 
User-agent: Surfbot 
User-agent: tAkeOut 
User-agent: Teleport\ Pro 
User-agent: VoidEYE 
User-agent: Web\ Image\ Collector 
User-agent: Web\ Sucker 
User-agent: WebAuto 
User-agent: WebCopier 
User-agent: WebFetch 
User-agent: WebGo\ IS 
User-agent: WebLeacher 
User-agent: WebReaper 
User-agent: WebSauger 
User-agent: Website\ eXtractor 
User-agent: Website\ Quester 
User-agent: WebStripper 
User-agent: WebWhacker 
User-agent: WebZIP 
User-agent: Wget 
User-agent: Widow 
User-agent: WWWOFFLE 
User-agent: Xaldon\ WebSpider 
User-agent: Zeus
Disallow: /
​
To answer your second question, no - once you have set Ahrefs to blocked on your website - data pertaining to your website will not show up on Ahrefs.​
​
You can still use Ahrefs to research your niche, but no website information will be available.​
 
BlogPro said:
Both Ahrefs and Semrush respect robots.txt entries. More details on their bots can be found at​
​
Ahrefs - https://ahrefs.com/robot​
Semrush - https://www.semrush.com/bot/​
​
To block Ahrefs, your robots.txt should look like​
​
Code: 
User-agent: AhrefsBot[/INDENT][/INDENT]
[INDENT=2][INDENT=2]Disallow: /
​
To block Semrush, your robots.txt should look like​
​
Code: 
User-agent: SemrushBot[/INDENT][/INDENT]
[INDENT=2][INDENT=2]Disallow: /
​
Moz uses Rogerbot and Dotbot to crawl. Former for site analysis, the latter for Link index.​
​
Other services use their own custom bots.​
​
Here's a list to ban the most common crawlers -​
​
Code: 
[/INDENT][/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Rogerbot [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Exabot [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: MJ12bot [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Dotbot [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Gigabot [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: AhrefsBot [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: BlackWidow[/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: ChinaClaw [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Custo [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: DISCo [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Download\ Demon [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: eCatch [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: EirGrabber [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: EmailSiphon [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: EmailWolf [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Express\ WebPictures [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: ExtractorPro [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: EyeNetIE [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: FlashGet [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: GetRight [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: GetWeb! [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Go!Zilla [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Go-Ahead-Got-It [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: GrabNet [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Grafula [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: HMView [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: HTTrack [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Image\ Stripper [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Image\ Sucker [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Indy\ Library[/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: InterGET [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Internet\ Ninja [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: JetCar [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: JOC\ Web\ Spider [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: larbin [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: LeechFTP [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Mass\ Downloader [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: MIDown\ tool [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Mister\ PiX [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Navroad [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: NearSite [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: NetAnts [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: NetSpider [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Net\ Vampire [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: NetZIP [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Octopus [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Offline\ Explorer [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Offline\ Navigator [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: PageGrabber [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Papa\ Foto [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: pavuk [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: pcBrowser [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: RealDownload [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: ReGet [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: SiteSnagger [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: SmartDownload [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: SuperBot [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: SuperHTTP [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Surfbot [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: tAkeOut [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Teleport\ Pro [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: VoidEYE [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Web\ Image\ Collector [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Web\ Sucker [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: WebAuto [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: WebCopier [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: WebFetch [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: WebGo\ IS [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: WebLeacher [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: WebReaper [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: WebSauger [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Website\ eXtractor [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Website\ Quester [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: WebStripper [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: WebWhacker [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: WebZIP [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Wget [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Widow [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: WWWOFFLE [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Xaldon\ WebSpider [/INDENT][/INDENT]
[INDENT=2][INDENT=2]User-agent: Zeus[/INDENT][/INDENT]
[INDENT=2][INDENT=2]Disallow: /[/INDENT][/INDENT]
[INDENT=2][INDENT=2][/INDENT][/INDENT]
[INDENT=2][INDENT=2][INDENT=2]
​
To answer your second question, no - once you have set Ahrefs to blocked on your website - data pertaining to your website will not show up on Ahrefs.​
​
You can still use Ahrefs to research your niche, but no website information will be available.​
Click to expand...
Thank you so much for this list mate.
 
BlogPro said:
Both Ahrefs and Semrush respect robots.txt entries. More details on their bots can be found at​
​
Ahrefs - https://ahrefs.com/robot​
Semrush - https://www.semrush.com/bot/​
​
To block Ahrefs, your robots.txt should look like​
​
Rich (BB code): 
User-agent: AhrefsBot
Disallow: /
​
To block Semrush, your robots.txt should look like​
​
Rich (BB code): 
User-agent: SemrushBot
Disallow: /
​
Moz uses Rogerbot and Dotbot to crawl. Former for site analysis, the latter for Link index.​
​
Other services use their own custom bots.​
​
Here's a list to ban the most common crawlers -​
​
Code: 
User-agent: Rogerbot
User-agent: Exabot
User-agent: MJ12bot
User-agent: Dotbot
User-agent: Gigabot
User-agent: AhrefsBot
User-agent: BlackWidow
User-agent: ChinaClaw
User-agent: Custo
User-agent: DISCo
User-agent: Download\ Demon
User-agent: eCatch
User-agent: EirGrabber
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: Express\ WebPictures
User-agent: ExtractorPro
User-agent: EyeNetIE
User-agent: FlashGet
User-agent: GetRight
User-agent: GetWeb!
User-agent: Go!Zilla
User-agent: Go-Ahead-Got-It
User-agent: GrabNet
User-agent: Grafula
User-agent: HMView
User-agent: HTTrack
User-agent: Image\ Stripper
User-agent: Image\ Sucker
User-agent: Indy\ Library
User-agent: InterGET
User-agent: Internet\ Ninja
User-agent: JetCar
User-agent: JOC\ Web\ Spider
User-agent: larbin
User-agent: LeechFTP
User-agent: Mass\ Downloader
User-agent: MIDown\ tool
User-agent: Mister\ PiX
User-agent: Navroad
User-agent: NearSite
User-agent: NetAnts
User-agent: NetSpider
User-agent: Net\ Vampire
User-agent: NetZIP
User-agent: Octopus
User-agent: Offline\ Explorer
User-agent: Offline\ Navigator
User-agent: PageGrabber
User-agent: Papa\ Foto
User-agent: pavuk
User-agent: pcBrowser
User-agent: RealDownload
User-agent: ReGet
User-agent: SiteSnagger
User-agent: SmartDownload
User-agent: SuperBot
User-agent: SuperHTTP
User-agent: Surfbot
User-agent: tAkeOut
User-agent: Teleport\ Pro
User-agent: VoidEYE
User-agent: Web\ Image\ Collector
User-agent: Web\ Sucker
User-agent: WebAuto
User-agent: WebCopier
User-agent: WebFetch
User-agent: WebGo\ IS
User-agent: WebLeacher
User-agent: WebReaper
User-agent: WebSauger
User-agent: Website\ eXtractor
User-agent: Website\ Quester
User-agent: WebStripper
User-agent: WebWhacker
User-agent: WebZIP
User-agent: Wget
User-agent: Widow
User-agent: WWWOFFLE
User-agent: Xaldon\ WebSpider
User-agent: Zeus
Disallow: /
​
To answer your second question, no - once you have set Ahrefs to blocked on your website - data pertaining to your website will not show up on Ahrefs.​
​
You can still use Ahrefs to research your niche, but no website information will be available.​
Click to expand...
One thing, blocking those bots won't affect google index in any way right?
 
Is this correct code to block all Ahrefs IP bot in .htaccess file?

SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>
Order Allow,Deny
Deny from 51.222.253.0
Deny from 51.222.253.1
Deny from 51.222.253.2
Deny from 51.222.253.3
Deny from 51.222.253.4
Deny from 51.222.253.5
Deny from 51.222.253.6
Deny from 51.222.253.7
Deny from 51.222.253.8
Deny from 51.222.253.9
Deny from 51.222.253.10
Deny from 51.222.253.11
Deny from 51.222.253.12
Deny from 51.222.253.13
Deny from 51.222.253.14
Deny from 51.222.253.15
Deny from 51.222.253.16
Deny from 51.222.253.17
Deny from 51.222.253.18
Deny from 51.222.253.19
Deny from 51.222.253.20
Deny from 51.222.253.21
Deny from 51.222.253.22
Deny from 51.222.253.23
Deny from 51.222.253.24
Deny from 51.222.253.25
Deny from 51.222.253.26
Deny from 51.222.253.27
Deny from 51.222.253.28
Deny from 51.222.253.29
Deny from 51.222.253.30
Deny from 51.222.253.31
Deny from 51.222.253.32
Deny from 51.222.253.33
Deny from 51.222.253.34
Deny from 51.222.253.35
Deny from 51.222.253.36
Deny from 51.222.253.37
Deny from 51.222.253.38
Deny from 51.222.253.39
Deny from 51.222.253.40
Deny from 51.222.253.41
Deny from 51.222.253.42
Deny from 51.222.253.43
Deny from 51.222.253.44
Deny from 51.222.253.45
Deny from 51.222.253.46
Deny from 51.222.253.47
Deny from 51.222.253.48
Deny from 51.222.253.49
Deny from 51.222.253.50
Deny from 51.222.253.51
Deny from 51.222.253.52
Deny from 51.222.253.53
Deny from 51.222.253.54
Deny from 51.222.253.55
Deny from 51.222.253.56
Deny from 51.222.253.57
Deny from 51.222.253.58
Deny from 51.222.253.59
Deny from 51.222.253.60
Deny from 51.222.253.61
Deny from 51.222.253.62
Deny from 51.222.253.63
Deny from 168.119.64.245
Deny from 168.119.64.246
Deny from 168.119.64.247
Deny from 168.119.64.248
Deny from 168.119.64.249
Deny from 168.119.64.250
Deny from 168.119.64.251
Deny from 168.119.64.252
Deny from 168.119.64.253
Deny from 168.119.64.254
Deny from 168.119.65.107
Deny from 168.119.65.108
Deny from 168.119.65.109
Deny from 168.119.65.110
Deny from 168.119.65.111
Deny from 168.119.65.112
Deny from 168.119.65.113
Deny from 168.119.65.114
Deny from 168.119.65.115
Deny from 168.119.65.116
Deny from 168.119.65.117
Deny from 168.119.65.118
Deny from 168.119.65.119
Deny from 168.119.65.120
Deny from 168.119.65.121
Deny from 168.119.65.122
Deny from 168.119.65.123
Deny from 168.119.65.124
Deny from 168.119.65.125
Deny from 168.119.65.126
Deny from 168.119.65.43
Deny from 168.119.65.44
Deny from 168.119.65.45
Deny from 168.119.65.46
Deny from 168.119.65.47
Deny from 168.119.65.48
Deny from 168.119.65.49
Deny from 168.119.65.50
Deny from 168.119.65.51
Deny from 168.119.65.52
Deny from 168.119.65.53
Deny from 168.119.65.54
Deny from 168.119.65.55
Deny from 168.119.65.56
Deny from 168.119.65.57
Deny from 168.119.65.58
Deny from 168.119.65.59
Deny from 168.119.65.60
Deny from 168.119.65.61
Deny from 168.119.65.62
Deny from 168.119.68.117
Deny from 168.119.68.118
Deny from 168.119.68.119
Deny from 168.119.68.120
Deny from 168.119.68.121
Deny from 168.119.68.122
Deny from 168.119.68.123
Deny from 168.119.68.124
Deny from 168.119.68.125
Deny from 168.119.68.126
Deny from 168.119.68.171
Deny from 168.119.68.172
Deny from 168.119.68.173
Deny from 168.119.68.174
Deny from 168.119.68.175
Deny from 168.119.68.176
Deny from 168.119.68.177
Deny from 168.119.68.178
Deny from 168.119.68.179
Deny from 168.119.68.180
Deny from 168.119.68.181
Deny from 168.119.68.182
Deny from 168.119.68.183
Deny from 168.119.68.184
Deny from 168.119.68.185
Deny from 168.119.68.186
Deny from 168.119.68.187
Deny from 168.119.68.188
Deny from 168.119.68.189
Deny from 168.119.68.190
Deny from 168.119.68.235
Deny from 168.119.68.236
Deny from 168.119.68.237
Deny from 168.119.68.238
Deny from 168.119.68.239
Deny from 168.119.68.240
Deny from 168.119.68.241
Deny from 168.119.68.242
Deny from 168.119.68.243
Deny from 168.119.68.244
Deny from 168.119.68.245
Deny from 168.119.68.246
Deny from 168.119.68.247
Deny from 168.119.68.248
Deny from 168.119.68.249
Deny from 168.119.68.250
Deny from 168.119.68.251
Deny from 168.119.68.252
Deny from 168.119.68.253
Deny from 168.119.68.254
Allow from all
Click to expand...
 
I did that with cloudfalre. And i didt that not because of competition but because sotimes these bots go mental and start making 1000`s of requests a minute... You can just go to cloudflare wirewall and block their ASN
 
do it on cloudflare (inside firewall settings)
do it on htaccess file
do it on your cpane/directadmin csf / cpguard firewall settings.
 
mostfurious said:
I did that with cloudfalre. And i didt that not because of competition but because sotimes these bots go mental and start making 1000`s of requests a minute... You can just go to cloudflare wirewall and block their ASN
Click to expand...

jennykumar said:
do it on cloudflare (inside firewall settings)
do it on htaccess file
do it on your cpane/directadmin csf / cpguard firewall settings.
Click to expand...
Can someone post the CF firewall settings to block all except GoogleBot. Most settings that I search on the web have to add each bot separately to the rules as shown below. Most of the bots still land up as we don't have a complete list of the bots and they keep changing the names.
 

Attachments

  • block-seo-crawlers-using-cloudflare-868x1024.jpg
    block-seo-crawlers-using-cloudflare-868x1024.jpg
    110.5 KB · Views: 64
BlogPro said:
Both Ahrefs and Semrush respect robots.txt entries. More details on their bots can be found at​
​
​
To block Ahrefs, your robots.txt should look like​
​
Rich (BB code): 
User-agent: AhrefsBot
Disallow: /
​
To block Semrush, your robots.txt should look like​
​
Rich (BB code): 
User-agent: SemrushBot
Disallow: /
​
Moz uses Rogerbot and Dotbot to crawl. Former for site analysis, the latter for Link index.​
​
Other services use their own custom bots.​
​
Here's a list to ban the most common crawlers -​
​
Code: 
User-agent: Rogerbot
User-agent: Exabot
User-agent: MJ12bot
User-agent: Dotbot
User-agent: Gigabot
User-agent: AhrefsBot
User-agent: BlackWidow
User-agent: ChinaClaw
User-agent: Custo
User-agent: DISCo
User-agent: Download\ Demon
User-agent: eCatch
User-agent: EirGrabber
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: Express\ WebPictures
User-agent: ExtractorPro
User-agent: EyeNetIE
User-agent: FlashGet
User-agent: GetRight
User-agent: GetWeb!
User-agent: Go!Zilla
User-agent: Go-Ahead-Got-It
User-agent: GrabNet
User-agent: Grafula
User-agent: HMView
User-agent: HTTrack
User-agent: Image\ Stripper
User-agent: Image\ Sucker
User-agent: Indy\ Library
User-agent: InterGET
User-agent: Internet\ Ninja
User-agent: JetCar
User-agent: JOC\ Web\ Spider
User-agent: larbin
User-agent: LeechFTP
User-agent: Mass\ Downloader
User-agent: MIDown\ tool
User-agent: Mister\ PiX
User-agent: Navroad
User-agent: NearSite
User-agent: NetAnts
User-agent: NetSpider
User-agent: Net\ Vampire
User-agent: NetZIP
User-agent: Octopus
User-agent: Offline\ Explorer
User-agent: Offline\ Navigator
User-agent: PageGrabber
User-agent: Papa\ Foto
User-agent: pavuk
User-agent: pcBrowser
User-agent: RealDownload
User-agent: ReGet
User-agent: SiteSnagger
User-agent: SmartDownload
User-agent: SuperBot
User-agent: SuperHTTP
User-agent: Surfbot
User-agent: tAkeOut
User-agent: Teleport\ Pro
User-agent: VoidEYE
User-agent: Web\ Image\ Collector
User-agent: Web\ Sucker
User-agent: WebAuto
User-agent: WebCopier
User-agent: WebFetch
User-agent: WebGo\ IS
User-agent: WebLeacher
User-agent: WebReaper
User-agent: WebSauger
User-agent: Website\ eXtractor
User-agent: Website\ Quester
User-agent: WebStripper
User-agent: WebWhacker
User-agent: WebZIP
User-agent: Wget
User-agent: Widow
User-agent: WWWOFFLE
User-agent: Xaldon\ WebSpider
User-agent: Zeus
Disallow: /
​
To answer your second question, no - once you have set Ahrefs to blocked on your website - data pertaining to your website will not show up on Ahrefs.​
​
You can still use Ahrefs to research your niche, but no website information will be available.​
Click to expand...
Thanks bro for this list!
 
BlogPro said:
Both Ahrefs and Semrush respect robots.txt entries. More details on their bots can be found at​
​
​
To block Ahrefs, your robots.txt should look like​
​
Rich (BB code): 
User-agent: AhrefsBot
Disallow: /
​
To block Semrush, your robots.txt should look like​
​
Rich (BB code): 
User-agent: SemrushBot
Disallow: /
​
Moz uses Rogerbot and Dotbot to crawl. Former for site analysis, the latter for Link index.​
​
Other services use their own custom bots.​
​
Here's a list to ban the most common crawlers -​
​
Code: 
User-agent: Rogerbot
User-agent: Exabot
User-agent: MJ12bot
User-agent: Dotbot
User-agent: Gigabot
User-agent: AhrefsBot
User-agent: BlackWidow
User-agent: ChinaClaw
User-agent: Custo
User-agent: DISCo
User-agent: Download\ Demon
User-agent: eCatch
User-agent: EirGrabber
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: Express\ WebPictures
User-agent: ExtractorPro
User-agent: EyeNetIE
User-agent: FlashGet
User-agent: GetRight
User-agent: GetWeb!
User-agent: Go!Zilla
User-agent: Go-Ahead-Got-It
User-agent: GrabNet
User-agent: Grafula
User-agent: HMView
User-agent: HTTrack
User-agent: Image\ Stripper
User-agent: Image\ Sucker
User-agent: Indy\ Library
User-agent: InterGET
User-agent: Internet\ Ninja
User-agent: JetCar
User-agent: JOC\ Web\ Spider
User-agent: larbin
User-agent: LeechFTP
User-agent: Mass\ Downloader
User-agent: MIDown\ tool
User-agent: Mister\ PiX
User-agent: Navroad
User-agent: NearSite
User-agent: NetAnts
User-agent: NetSpider
User-agent: Net\ Vampire
User-agent: NetZIP
User-agent: Octopus
User-agent: Offline\ Explorer
User-agent: Offline\ Navigator
User-agent: PageGrabber
User-agent: Papa\ Foto
User-agent: pavuk
User-agent: pcBrowser
User-agent: RealDownload
User-agent: ReGet
User-agent: SiteSnagger
User-agent: SmartDownload
User-agent: SuperBot
User-agent: SuperHTTP
User-agent: Surfbot
User-agent: tAkeOut
User-agent: Teleport\ Pro
User-agent: VoidEYE
User-agent: Web\ Image\ Collector
User-agent: Web\ Sucker
User-agent: WebAuto
User-agent: WebCopier
User-agent: WebFetch
User-agent: WebGo\ IS
User-agent: WebLeacher
User-agent: WebReaper
User-agent: WebSauger
User-agent: Website\ eXtractor
User-agent: Website\ Quester
User-agent: WebStripper
User-agent: WebWhacker
User-agent: WebZIP
User-agent: Wget
User-agent: Widow
User-agent: WWWOFFLE
User-agent: Xaldon\ WebSpider
User-agent: Zeus
Disallow: /
​
To answer your second question, no - once you have set Ahrefs to blocked on your website - data pertaining to your website will not show up on Ahrefs.​
​
You can still use Ahrefs to research your niche, but no website information will be available.​
Click to expand...


Do you know if there's a plugin to block all of these?
 
vin42 said:
Can someone post the CF firewall settings to block all except GoogleBot. Most settings that I search on the web have to add each bot separately to the rules as shown below. Most of the bots still land up as we don't have a complete list of the bots and they keep changing the names.
Click to expand...
You will get tired by adding all bots... instead, block them all and allow only the ones you want like so:
google.PNG

Note that Asnum i selected "DOES NOT EQUAL" thats important. Esentially this rule means if its a known bot (google, bing etc) and the asn IS NOT equal to 15169 (thats googles network), then block it.

google bot.PNG

Will this block every and all bots ? NO, you have to check in cloudflare from time to time. And block them manualy. I just block the ASN, the easiest way to deal with them. Ips, names can change but ASN generally stays the same.
 
mostfurious said:
I did that with cloudfalre. And i didt that not because of competition but because sotimes these bots go mental and start making 1000`s of requests a minute... You can just go to cloudflare wirewall and block their ASN
Click to expand...
These f&*^&in bots are driving me crazy and crashing my server.....I have blocked known bots excluding G and bing, however even G and bingbots are sending thousands of requests....I have now added crawl delay in bing webmaster but G doesn't seem to have that option.
 
vin42 said:
These f&*^&in bots are driving me crazy and crashing my server.....I have blocked known bots excluding G and bing, however even G and bingbots are sending thousands of requests....I have now added crawl delay in bing webmaster but G doesn't seem to have that option.
Click to expand...
I know what you mean.... i get hundreds of those a day from google aswell.... How many times a day do you need to request THE SAME js file for indexing ? Why would google need that in the first place ? That is why i have other rule for google that if it tries to access images (jpg, png), style sheets (css), javascript files (js) those requests get blocked. You can access what matters (content) but no stylesheets, javascript and images. That usually shaves around 600 reuqests on one website a day. All the websites are still ranking fine.
 
vin42 said:
These f&*^&in bots are driving me crazy and crashing my server.....I have blocked known bots excluding G and bing, however even G and bingbots are sending thousands of requests....I have now added crawl delay in bing webmaster but G doesn't seem to have that option.
Click to expand...
adding cloaker to your site should fix that issue
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top