can`t block Ahref bot?

Chris.seo

Newbie
Joined
Feb 12, 2021
Messages
17
Reaction score
5
Hi,

I want to block web crawler bots on some of my PBN`s.

As far as I know the best way to do it is from .htaccess.

I am using the following command, but it seems it doesn`t work and Ahref still detect the links from my PBN sites:

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} (ahrefsbot|mj12bot|rogerbot|exabot|dotbot|gigabot|semrush) [NC]
RewriteRule .* - [F,L]
</IfModule>

Could you please let me know what code do you use to block web crawler bots (Ahref, MOZ, Majestic, Semrush etc) and what is the best way to do it?

Thank you!
 

k09t4z

Newbie
Joined
Feb 18, 2022
Messages
1
Reaction score
0
for Ahref added this

Code:
 deny from 54.36.148.0/24
 deny from 54.36.149.0/24
 deny from 195.154.122.0/24
 deny from 195.154.123.0/24
 deny from 195.154.126.0/24
 deny from 195.154.127.0/24
 deny from 51.222.253.0/26

https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges
 

getivan

Banned - Multiple Rules Violations
Joined
Sep 1, 2017
Messages
379
Reaction score
441
I saw someone saying they blocked Ahrefs, and other tools, the other day, but then dropped their samples right into some tools, anyways. :eek:
Waste of time to block tools.
You can't stop @MisterF :rolleyes:
 

PetalSEO

BANNED
Joined
Oct 23, 2021
Messages
488
Reaction score
148
for Ahref added this

Code:
 deny from 54.36.148.0/24
 deny from 54.36.149.0/24
 deny from 195.154.122.0/24
 deny from 195.154.123.0/24
 deny from 195.154.126.0/24
 deny from 195.154.127.0/24
 deny from 51.222.253.0/26

https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges

Great , this would work over uk or any other . domains ?
 

Zod

Jr. VIP
Jr. VIP
Joined
Oct 12, 2019
Messages
206
Reaction score
86
Hey I know a solution for this.
If youn use cloudflare then block the ASN server ahrefs use and they will not be able to scan or crawl
worked for me
 

newslinkseo

Banned - Spam posting
Joined
Feb 17, 2022
Messages
73
Reaction score
29
So far there no one here has answered the question correctly.

You block ahrefs bot via your ROBOTS.TXT file

Please add this in your website robots.txt file

User-agent: AhrefsBot

Disallow:/



Basically any bot or agent you want to block you do this in robots.txt
 

Chris.seo

Newbie
Joined
Feb 12, 2021
Messages
17
Reaction score
5
So far there no one here has answered the question correctly.

You block ahrefs bot via your ROBOTS.TXT file

Please add this in your website robots.txt file

User-agent: AhrefsBot

Disallow:/



Basically any bot or agent you want to block you do this in robots.txt
Hm I thought .htaccess is better option since some aggressive bots might disregard the robots.txt?

Maybe it will not hurt to do both?
 

Chris.seo

Newbie
Joined
Feb 12, 2021
Messages
17
Reaction score
5
Hey I know a solution for this.
If youn use cloudflare then block the ASN server ahrefs use and they will not be able to scan or crawl
worked for me

Thanks, I have cloudflare on all of my PBN`s, I will check this option as well!
 

Mangamad

BANNED
Joined
Feb 16, 2022
Messages
27
Reaction score
10
for Ahref added this

Code:
 deny from 54.36.148.0/24
 deny from 54.36.149.0/24
 deny from 195.154.122.0/24
 deny from 195.154.123.0/24
 deny from 195.154.126.0/24
 deny from 195.154.127.0/24
 deny from 51.222.253.0/26

https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges
Try this it shall help , or hire an web developer for this .
 

fb3003

Jr. VIP
Jr. VIP
Joined
Aug 14, 2015
Messages
4,460
Reaction score
3,020
Website
bit.ly
So far there no one here has answered the question correctly.

You block ahrefs bot via your ROBOTS.TXT file

Please add this in your website robots.txt file

User-agent: AhrefsBot

Disallow:/



Basically any bot or agent you want to block you do this in robots.txt


i suggest to do this and report back.... ahref will still index you... they are using agents and IPs we don't know about
 

Zod

Jr. VIP
Jr. VIP
Joined
Oct 12, 2019
Messages
206
Reaction score
86
i suggest to do this and report back.... ahref will still index you... they are using agents and IPs we don't know about
I ahve told you the best answer already
check the thread


check the reply below
Hey I know a solution for this.
If youn use cloudflare then block the ASN server ahrefs use and they will not be able to scan or crawl
worked for me
 

newslinkseo

Banned - Spam posting
Joined
Feb 17, 2022
Messages
73
Reaction score
29
Hm I thought .htaccess is better option since some aggressive bots might disregard the robots.txt?

Maybe it will not hurt to do both?

It won't hurt to do both. But its not needed. Robots.txt rules over everything

Cloudflare, htaccess. All not required. And neither is the best solution. ROBOTS.TXT is the No1. config choice and the most simple. Therefore the best.

Just open up robots.txt add these two lines are you are done! No need for mucking about with ASNs. Its a pointless waste of time.

User-agent: AhrefsBot

Disallow:/
 

itz_styx

Jr. VIP
Jr. VIP
Joined
May 8, 2012
Messages
2,589
Reaction score
1,906
Website
argo-content.com
they claim to obey robots.txt and additionally you can block their IP ranges either via .htaccess or iptables for example.
here you can find more details including their current IP lists: https://ahrefs.com/robot
 
Top