Detecting PBN's Shower Thought...

If a typical telltale sign of a PBN is they block scraping bots like ahrefs from indexing their site. then would a good way to detect PBNs be to index every robots.txt and inspect sites blocking ahrefs?

I'm not sure if I'm allowed to reveal this info on Public forum but you don't just "index" the robots.txt file but try sending a http request with the user-agent : "AhrefsBot" and send another request with "GoogleBot". If you get 404 in the first request and 200 success in the 2nd, they're blocking the Ahrefs bot from crawling their site and you can conclude it as a PBN with a little manual inspection.

SpoonFeeder said:

I'm not sure if I'm allowed to reveal this info on Public forum but you don't just "index" the robots.txt file but try sending a http request with the user-agent : "AhrefsBot" and send another request with "GoogleBot". If you get 404 in the first request and 200 success in the 2nd, they're blocking the Ahrefs bot from crawling their site and you can conclude it as a PBN with a little manual inspection.

Click to expand...

Yep something like this was what I was thinking,

Now what if I was to automate that and start scraping the web and building an index of googlebot: on, ahrefs bot: off
Of course anyone who is doing the PBN right will allow bots and have all domains properly sanitized from each other but pretty sure I could write a script that could easily catch some fish

SpoonFeeder said:

I'm not sure if I'm allowed to reveal this info on Public forum but you don't just "index" the robots.txt file but try sending a http request with the user-agent : "AhrefsBot" and send another request with "GoogleBot". If you get 404 in the first request and 200 success in the 2nd, they're blocking the Ahrefs bot from crawling their site and you can conclude it as a PBN with a little manual inspection.

Click to expand...

that only works on cloakers that don't have proper checks in place and only use simple UA cloaking. also many legitimate sites block ahrefs and other bots they dont care about. ahrefs just takes up resources, but doesnt add any benefit to any website, so its safe to block. that being said, you can't just assume a site is part of a PBN, because it blocks ahrefs

scottledeuce said:

Yep something like this was what I was thinking,

Now what if I was to automate that and start scraping the web and building an index of googlebot: on, ahrefs bot: off
Of course anyone who is doing the PBN right will allow bots and have all domains properly sanitized from each other but pretty sure I could write a script that could easily catch some fish

Click to expand...

It's not that hard actually. I you REALLY want it, you can get it done in no time and catch a lot of catfishes along the line as 99% of the peoples who buy/build a PBN don't have a clue on how to hide/protect it from "smart" peoples like you.

itz_styx said:

that only works on cloakers that don't have proper checks in place and only use simple UA cloaking. also many legitimate sites block ahrefs and other bots they dont care about. ahrefs just takes up resources, but doesnt add any benefit to any website, so its safe to block. that being said, you can't just assume a site is part of a PBN, because it blocks ahrefs

Click to expand...

True. It's just a food for thought post and I wouldn't reveal the extensive list of strategies to detect PBN's and that too on a public post. That's just shooting oneself in the foot.

SpoonFeeder said:

I'm not sure if I'm allowed to reveal this info on Public forum but you don't just "index" the robots.txt file but try sending a http request with the user-agent : "AhrefsBot" and send another request with "GoogleBot". If you get 404 in the first request and 200 success in the 2nd, they're blocking the Ahrefs bot from crawling their site and you can conclude it as a PBN with a little manual inspection.

Click to expand...

Once you know it's a PBN what do you do? What is the next step?

SpoonFeeder said:

It's not that hard actually. I you REALLY want it, you can get it done in no time and catch a lot of catfishes along the line as 99% of the peoples who buy/build a PBN don't have a clue on how to hide/protect it from "smart" peoples like you.



True. It's just a food for thought post and I wouldn't reveal the extensive list of strategies to detect PBN's and that too on a public post. That's just shooting oneself in the foot.

Click to expand...

I been going down the rabbit hole on this and pretty much have it figured out,

SpoonFeeder I'd like to PM you something if you don't want to discuss it on open board. essentially its the same thing but done through hta.access and detection.

I've learnt thats the right way to hide your PBN but there are also signatures if not done right.?
am I barking up the right tree here ?

pbn have chaged know days how there made .

the new way is you dont hide nothink and build a strong website and point it to your money site and point to other web 2.0 website to look legit .

hiding pbn is old days tech.

google wont take down well made website with grate content and backlinks .

you make the website same as useall but you convince google it legit .

this way you got a pbn/website for ages in life.

blocking anythink can trigger problams these days.

a well made website in same fashion as pbn built ,with no blocking wont get ban if google convince it a real used website ...

Hiding PBN is a pretty low tech in my eyes and only likely to draw suspicion.
My personal method is to just airgap each one properly as if they are indiividual sites.

That said Im looking at a competitor who is not outranking me on a very competitive keyword with a landing page that has zero banklinks.
I can see they are not blocking crawlers to the homepage as they have a good B/L profile but my suspicion is they are blocking this one specific subdirectory completely and ranking it with links I'm powerless to see right now,

SO im using this as a learning exercise, Need to reverse engineer what they did and FOR SURE they did something to get this page where it is!