BubbleCorporation
Newbie
- May 11, 2025
- 21
- 9
Hello i am looking for a pbn tool to hide my pbn from ahrefs majestic moz etc etc looking for the robot.txt script
Looking for Script to hide PBN sites from crawlers
- Thread starter BubbleCorporation
- Start date
- Sep 17, 2008
- 6,037
- 3,973
I wouldn't bother. If there was a reliable and accurate way of doing this, the crawlers would see it as a sign that you are part of a PBN. I would leave it, not worth leaving that footprint imo.
- Aug 16, 2014
- 3,099
- 679
I can not help with tools or methods that hide PBNs from Ahrefs, Majestic, Moz, or similar services, or anything meant to cloak a network. But if you want, I can share safer, white hat SEO strategies that help you grow organically without risking your sites.
verismei_meint
Junior Member
- Nov 6, 2025
- 102
- 32
Use a robots.txt file to block all crawlers:
User-agent: *
Disallow: /
For stronger protection, use IP blocking in your .htaccess file.
User-agent: *
Disallow: /
For stronger protection, use IP blocking in your .htaccess file.
- Apr 3, 2009
- 567
- 299
To expand on this a bit:
robots.txt is useless for this - crawlers ignore it when they want to, and blocking everything is a footprint in itself.
The approach that works: server-level blocking based on IP ranges (Ahrefs, Moz, Majestic all publish their crawler IPs) combined with user-agent detection as a fallback. I wrote a script that checks both.
The gotcha is caching. If your host has server-side caching (LiteSpeed, Varnish, etc.) or installs caching plugins without telling you, the block never fires because the crawler gets served a cached page before your code runs. You need to either exclude the blocking logic from the cache or handle it at the edge (Cloudflare Workers, .htaccess, etc.).
Been running this on 800+ sites with no issues so far. Happy to share the approach if anyone wants details.
robots.txt is useless for this - crawlers ignore it when they want to, and blocking everything is a footprint in itself.
The approach that works: server-level blocking based on IP ranges (Ahrefs, Moz, Majestic all publish their crawler IPs) combined with user-agent detection as a fallback. I wrote a script that checks both.
The gotcha is caching. If your host has server-side caching (LiteSpeed, Varnish, etc.) or installs caching plugins without telling you, the block never fires because the crawler gets served a cached page before your code runs. You need to either exclude the blocking logic from the cache or handle it at the edge (Cloudflare Workers, .htaccess, etc.).
Been running this on 800+ sites with no issues so far. Happy to share the approach if anyone wants details.
- Apr 23, 2012
- 2,843
- 5,854
I wrote a post in another thread a few years ago on how to block ahrefs, semrush and a bunch of other bots using their user agents. Maybe you will find it useful
https://www.blackhatworld.com/seo/how-to-block-ahrefs-semrush-and-others.1392594/#post-15098905
Edit - You can also find an exhaustive list of bots that Cloudflare maintains here - https://radar.cloudflare.com/bots#verified-bots
https://www.blackhatworld.com/seo/how-to-block-ahrefs-semrush-and-others.1392594/#post-15098905
Edit - You can also find an exhaustive list of bots that Cloudflare maintains here - https://radar.cloudflare.com/bots#verified-bots
bint alkhandaq
Newbie
- Jun 4, 2025
- 8
- 2
You have to change the .htaccess. Specifically add code to that file to block a multitude of crawlers
bint alkhandaq
Newbie
- Jun 4, 2025
- 8
- 2
So would you recommend adding blocking code to the .htaccess to combat the caching? Also noob questions, how would you find out that the host has installed caching plugins w/o telling me. I control all my pbn domains via wordpress and install/remove plugins at my will. Is there anywhere else I should look?
- Jun 7, 2012
- 4,385
- 1,418
It's almost impossible to block all the GPT bots, even at the server level, your best bet would be using the Cloudflare option (paid at $25, I am not sure), but the AI training bots will still be able to crawl.
- Oct 25, 2011
- 1,649
- 933
It's totally possible to block bots from accessing the site, but the best way is to do so before a request even reaches the website.
Most of the time, you would need something in front of your website. As already mentioned, when your sites are cached (it depends on how), using htaccess may or may not work. It should work in most cases tho because htaccess rules are executed before your site's code is.
We have done multiple tests with our own websites to block Semrush and Ahrefs, and they have been very successful. One of the tests that we set up many years ago for some websites proved that it is possible to block those bots, and those websites haven't been indexed by Ahrefs or Semrush since then.
Most of the time, you would need something in front of your website. As already mentioned, when your sites are cached (it depends on how), using htaccess may or may not work. It should work in most cases tho because htaccess rules are executed before your site's code is.
We have done multiple tests with our own websites to block Semrush and Ahrefs, and they have been very successful. One of the tests that we set up many years ago for some websites proved that it is possible to block those bots, and those websites haven't been indexed by Ahrefs or Semrush since then.
lewis_martin
Junior Member
- Mar 9, 2026
- 103
- 30
what you’re trying to do doesn’t really work the way you think
You can block bots like Ahrefs/Majestic in robots.txt (e.g. User-agent: AhrefsBot Disallow: /), but that only stops them crawling your site not discovering your links elsewhere.
Even worse, if you block Googlebot too, your links won’t pass value because Google can’t see them 
So yeah there’s no “magic robots.txt script” to safely hide a PBN. Either Google sees the links (and risk it), or hides them (and they don’t work).
You can block bots like Ahrefs/Majestic in robots.txt (e.g. User-agent: AhrefsBot Disallow: /), but that only stops them crawling your site not discovering your links elsewhere.
Even worse, if you block Googlebot too, your links won’t pass value because Google can’t see them
So yeah there’s no “magic robots.txt script” to safely hide a PBN. Either Google sees the links (and risk it), or hides them (and they don’t work).
- Feb 28, 2019
- 7,428
- 7,865
Robots file just gives direction. It won't block anything. If ahrefs bot wants to crawl your website, it can ignore robots file and crawl it anyway. If you want to strictly block all such bots, you should block them at CDN or server level. AI can help you write code to add in htaccess file but you need clarity on what to block. Otherwise you might fail to pass link juice.
- Jun 15, 2021
- 289
- 23
robots txt file is the standard way to talk to bots many aggressive SEO crawlers can ignore it for a pbn the htaccess method is much more effective because it blocks them at the server level before they even see your content
laur.laurix
Elite Member
- May 8, 2013
- 2,063
- 1,871
Blocking seo crawlers is a strong signal for anti-spam G algo......G crawls the web using randoms IP classes along with it's official G bot....I was able to map a lot of G unofficial IP addresses but they rotate them a lot and it becomes worthless.....so it's not advisable to that.
bint alkhandaq
Newbie
- Jun 4, 2025
- 8
- 2
The tests you have done with your sites, did you block the SEO crawler bots with the htaccess? Or did you create a new way to prevent them from crawling your site?
Thank you for your response btw.
- Oct 25, 2011
- 1,649
- 933
No, we use our own firewall, but htaccess should be effective as well.
vierasen
Regular Member
- Jan 26, 2016
- 247
- 73
.htaccess is the right layer — it runs at the Apache/nginx level before PHP ever loads, so WordPress caching plugins, LiteSpeed cache, and similar tools are completely bypassed. The block fires first. That's the correct approach for this exact reason.
For detecting host-side caching you didn't configure: run curl -I https://yourdomain.com and look at the response headers. X-Cache, X-LiteSpeed-Cache, X-Varnish, or CF-Cache-Status headers showing up when you didn't set them means your host is caching at the server level. LiteSpeed in particular is commonly enabled by default on shared hosting without being visible in your plugin list — it runs outside WordPress entirely.
Also worth checking: your host control panel (cPanel/Plesk) often has a separate "LiteSpeed Web Cache" or "Caching" section that's completely separate from WP-Admin. If it's there and active, your .htaccess rewrite rules still run first, so you're fine — just worth knowing it's there.
For detecting host-side caching you didn't configure: run curl -I https://yourdomain.com and look at the response headers. X-Cache, X-LiteSpeed-Cache, X-Varnish, or CF-Cache-Status headers showing up when you didn't set them means your host is caching at the server level. LiteSpeed in particular is commonly enabled by default on shared hosting without being visible in your plugin list — it runs outside WordPress entirely.
Also worth checking: your host control panel (cPanel/Plesk) often has a separate "LiteSpeed Web Cache" or "Caching" section that's completely separate from WP-Admin. If it's there and active, your .htaccess rewrite rules still run first, so you're fine — just worth knowing it's there.
bint alkhandaq
Newbie
- Jun 4, 2025
- 8
- 2
Thank you. I'm super new at this, so a lot of this went over my head. But I'll start to look into it so I can understand what's going on and then understand your post. Thanks!
- Oct 25, 2011
- 1,649
- 933
If you are new, then you shouldn't worry about blocking bots. You absolutely don't have to block bots to run a successful PBN.