can`t block Ahref bot?

Chris.seo · Feb 18, 2022

Hi,

I want to block web crawler bots on some of my PBN`s.

As far as I know the best way to do it is from .htaccess.

I am using the following command, but it seems it doesn`t work and Ahref still detect the links from my PBN sites:

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} (ahrefsbot|mj12bot|rogerbot|exabot|dotbot|gigabot|semrush) [NC]
RewriteRule .* - [F,L]
</IfModule>

Could you please let me know what code do you use to block web crawler bots (Ahref, MOZ, Majestic, Semrush etc) and what is the best way to do it?

Thank you!

k09t4z · Feb 19, 2022

for Ahref added this

Code:

 deny from 54.36.148.0/24
 deny from 54.36.149.0/24
 deny from 195.154.122.0/24
 deny from 195.154.123.0/24
 deny from 195.154.126.0/24
 deny from 195.154.127.0/24
 deny from 51.222.253.0/26

https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges

LinkJar · Feb 19, 2022

Check with this - https://www.matthewwoodward.co.uk/seo/ahrefsbot/ - It shows you What is Ahrefsbot and How To Block it with more detailed steps .

Hope this Helps !

getivan · Feb 19, 2022

I saw someone saying they blocked Ahrefs, and other tools, the other day, but then dropped their samples right into some tools, anyways.

Waste of time to block tools.
You can't stop @MisterF

fb3003 · Feb 19, 2022

getivan said:
I saw someone saying they blocked Ahrefs, and other tools, the other day, but then dropped their samples right into some tools, anyways.
Waste of time to block tools.
You can't stop @MisterF

I agree - i think they use other IPs and bots to find all the links....

PetalSEO · Feb 19, 2022

k09t4z said:
for Ahref added this

Code:

deny from 54.36.148.0/24 deny from 54.36.149.0/24 deny from 195.154.122.0/24 deny from 195.154.123.0/24 deny from 195.154.126.0/24 deny from 195.154.127.0/24 deny from 51.222.253.0/26

https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges

Great , this would work over uk or any other . domains ?

fb3003 · Feb 19, 2022

PetalSEO said:
Great , this would work over uk or any other . domains ?

my suggestion, do this:
hit windows + r
type cmd
type format c:\

Zod · Feb 19, 2022

Hey I know a solution for this.
If youn use cloudflare then block the ASN server ahrefs use and they will not be able to scan or crawl
worked for me

newslinkseo · Feb 19, 2022

So far there no one here has answered the question correctly.

You block ahrefs bot via your ROBOTS.TXT file

Please add this in your website robots.txt file

User-agent: AhrefsBot

Disallow:/

Basically any bot or agent you want to block you do this in robots.txt

Chris.seo · Feb 21, 2022

newslinkseo said:
So far there no one here has answered the question correctly.

You block ahrefs bot via your ROBOTS.TXT file

Please add this in your website robots.txt file

User-agent: AhrefsBot

Disallow:/

Basically any bot or agent you want to block you do this in robots.txt

Hm I thought .htaccess is better option since some aggressive bots might disregard the robots.txt?

Maybe it will not hurt to do both?

Chris.seo · Feb 21, 2022

Karanjot69 said:
Hey I know a solution for this.
If youn use cloudflare then block the ASN server ahrefs use and they will not be able to scan or crawl
worked for me

Thanks, I have cloudflare on all of my PBN`s, I will check this option as well!

Mangamad · Feb 21, 2022

k09t4z said:
for Ahref added this

Code:

deny from 54.36.148.0/24 deny from 54.36.149.0/24 deny from 195.154.122.0/24 deny from 195.154.123.0/24 deny from 195.154.126.0/24 deny from 195.154.127.0/24 deny from 51.222.253.0/26

https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges

Try this it shall help , or hire an web developer for this .

fb3003 · Feb 21, 2022

newslinkseo said:
So far there no one here has answered the question correctly.

You block ahrefs bot via your ROBOTS.TXT file

Please add this in your website robots.txt file

User-agent: AhrefsBot

Disallow:/

Basically any bot or agent you want to block you do this in robots.txt

i suggest to do this and report back.... ahref will still index you... they are using agents and IPs we don't know about

Zod · Feb 21, 2022

fb3003 said:
i suggest to do this and report back.... ahref will still index you... they are using agents and IPs we don't know about

I ahve told you the best answer already
check the thread

check the reply below

Karanjot69 said:
Hey I know a solution for this.
If youn use cloudflare then block the ASN server ahrefs use and they will not be able to scan or crawl
worked for me

newslinkseo · Feb 21, 2022

Chris.seo said:
Hm I thought .htaccess is better option since some aggressive bots might disregard the robots.txt?

Maybe it will not hurt to do both?

It won't hurt to do both. But its not needed. Robots.txt rules over everything

Cloudflare, htaccess. All not required. And neither is the best solution. ROBOTS.TXT is the No1. config choice and the most simple. Therefore the best.

Just open up robots.txt add these two lines are you are done! No need for mucking about with ASNs. Its a pointless waste of time.

User-agent: AhrefsBot

Disallow:/

itz_styx · Feb 21, 2022

they claim to obey robots.txt and additionally you can block their IP ranges either via .htaccess or iptables for example.
here you can find more details including their current IP lists: https://ahrefs.com/robot

DenmarkTR · Mar 3, 2022

@Karanjot69 bro pls add me mein discord Mody#5988

can`t block Ahref bot?

Chris.seo

Newbie

k09t4z

Newbie

LinkJar

Jr. VIP

getivan

Banned - Multiple Rules Violations

fb3003

Jr. VIP

PetalSEO

BANNED

fb3003

Jr. VIP

Zod

Jr. VIP

newslinkseo

Banned - Spam posting

Chris.seo

Newbie

Chris.seo

Newbie

Mangamad

BANNED

fb3003

Jr. VIP

Zod

Jr. VIP

newslinkseo

Banned - Spam posting

itz_styx

Jr. VIP

DenmarkTR

Newbie

Main Menu

Marketplace

Making Money

BlackHatWorld

Other