can`t block Ahref bot?

[Chris.seo]

Hi,

I want to block web crawler bots on some of my PBN`s.

As far as I know the best way to do it is from .htaccess.

I am using the following command, but it seems it doesn`t work and Ahref still detect the links from my PBN sites:

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} (ahrefsbot|mj12bot|rogerbot|exabot|dotbot|gigabot|semrush) [NC]
RewriteRule .* - [F,L]
</IfModule>

Could you please let me know what code do you use to block web crawler bots (Ahref, MOZ, Majestic, Semrush etc) and what is the best way to do it?

Thank you!

 

[k09t4z]

for Ahref added this

 deny from 54.36.148.0/24
 deny from 54.36.149.0/24
 deny from 195.154.122.0/24
 deny from 195.154.123.0/24
 deny from 195.154.126.0/24
 deny from 195.154.127.0/24
 deny from 51.222.253.0/26

https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges

 

[LinkJar]

Check with this - https://www.matthewwoodward.co.uk/seo/ahrefsbot/ - It shows you What is Ahrefsbot and How To Block it with more detailed steps .

Hope this Helps !

 

[getivan]

I saw someone saying they blocked Ahrefs, and other tools, the other day, but then dropped their samples right into some tools, anyways.
Waste of time to block tools.
You can't stop @MisterF

 

[fb3003]

I saw someone saying they blocked Ahrefs, and other tools, the other day, but then dropped their samples right into some tools, anyways.
Waste of time to block tools.
You can't stop @MisterF

I agree - i think they use other IPs and bots to find all the links....

 

[PetalSEO]

for Ahref added this

 deny from 54.36.148.0/24
 deny from 54.36.149.0/24
 deny from 195.154.122.0/24
 deny from 195.154.123.0/24
 deny from 195.154.126.0/24
 deny from 195.154.127.0/24
 deny from 51.222.253.0/26

https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges

Great , this would work over uk or any other . domains ?

 

[fb3003]

Great , this would work over uk or any other . domains ?

my suggestion, do this:
hit windows + r
type cmd
type format c:\

 

[Zod]

Hey I know a solution for this.
If youn use cloudflare then block the ASN server ahrefs use and they will not be able to scan or crawl
worked for me

 

[newslinkseo]

So far there no one here has answered the question correctly.

You block ahrefs bot via your ROBOTS.TXT file

Please add this in your website robots.txt file

User-agent: AhrefsBot

Disallow:/



Basically any bot or agent you want to block you do this in robots.txt

 

[Chris.seo]

So far there no one here has answered the question correctly.

You block ahrefs bot via your ROBOTS.TXT file

Please add this in your website robots.txt file

User-agent: AhrefsBot

Disallow:/



Basically any bot or agent you want to block you do this in robots.txt

Hm I thought .htaccess is better option since some aggressive bots might disregard the robots.txt?

Maybe it will not hurt to do both?

 

[Chris.seo]

Hey I know a solution for this.
If youn use cloudflare then block the ASN server ahrefs use and they will not be able to scan or crawl
worked for me

Thanks, I have cloudflare on all of my PBN`s, I will check this option as well!

 

[Mangamad]

for Ahref added this

 deny from 54.36.148.0/24
 deny from 54.36.149.0/24
 deny from 195.154.122.0/24
 deny from 195.154.123.0/24
 deny from 195.154.126.0/24
 deny from 195.154.127.0/24
 deny from 51.222.253.0/26

https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges

Try this it shall help , or hire an web developer for this .

 

[fb3003]

So far there no one here has answered the question correctly.

You block ahrefs bot via your ROBOTS.TXT file

Please add this in your website robots.txt file

User-agent: AhrefsBot

Disallow:/



Basically any bot or agent you want to block you do this in robots.txt

i suggest to do this and report back.... ahref will still index you... they are using agents and IPs we don't know about

 

[Zod]

i suggest to do this and report back.... ahref will still index you... they are using agents and IPs we don't know about

I ahve told you the best answer already
check the thread


check the reply below

Hey I know a solution for this.
If youn use cloudflare then block the ASN server ahrefs use and they will not be able to scan or crawl
worked for me

 

[newslinkseo]

Hm I thought .htaccess is better option since some aggressive bots might disregard the robots.txt?

Maybe it will not hurt to do both?

It won't hurt to do both. But its not needed. Robots.txt rules over everything

Cloudflare, htaccess. All not required. And neither is the best solution. ROBOTS.TXT is the No1. config choice and the most simple. Therefore the best.

Just open up robots.txt add these two lines are you are done! No need for mucking about with ASNs. Its a pointless waste of time.

User-agent: AhrefsBot

Disallow:/

 

[itz_styx]

they claim to obey robots.txt and additionally you can block their IP ranges either via .htaccess or iptables for example.
here you can find more details including their current IP lists: https://ahrefs.com/robot

 

[DenmarkTR]

@Karanjot69 bro pls add me mein discord Mody#5988