Dropl
Senior Member
- Feb 17, 2012
- 926
- 144
Exactly! Google crawlers probably have some sort checks for these kinds of things. After all, they read robots.txt all day long
Why Do PBNs owners block ahrefs bots
- Thread starter ahmedtarek8572
- Start date
True Damage
Registered Member
- Apr 30, 2021
- 53
- 38
Well any business with a badly coded site would block crawlers.
hideath
Elite Member
- Jun 16, 2019
- 6,620
- 7,350
Most of the time it doesn't actually cause any problems neither devaluing the network,
You can always cloak the links to show them only to google crawlers which would eliminate such a footprint,
In both cases leaving it exposed would be much worst as it makes it a piece of cake to get reported and taken down
- Apr 4, 2009
- 13,911
- 70,968
As others above said, blocking competitors to find their links.
One way of finding competitors links on pbn´s is to use Google Alerts and monitor keywords, I do that locally with a smaller market, whenever a competitor publishes an article with a keyword I´m monitoring, sooner or later I´ll get an alert. Not a 100% method, but I found a lot of wierd and great sites that way .. Try it, it´s free .. and they guaranteed don´t block Google .. =)
One way of finding competitors links on pbn´s is to use Google Alerts and monitor keywords, I do that locally with a smaller market, whenever a competitor publishes an article with a keyword I´m monitoring, sooner or later I´ll get an alert. Not a 100% method, but I found a lot of wierd and great sites that way .. Try it, it´s free .. and they guaranteed don´t block Google .. =)
fb3003
Elite Member
- Aug 14, 2015
- 5,035
- 3,500
next question that you might get: what is a tom, dick and harry?
- Sep 7, 2017
- 7,187
- 6,435
In case anyone intends to ask; Tom, Dick and Harry are a triplet working for Google. The interesting thing about them is they were born for the sole purpose of fishing out errant SEO. They always work for Google by the way, no other company can afford to employ them.
alexono1
Regular Member
- Sep 14, 2012
- 229
- 138
Real seo's are as paranoid as hell... So they will try to hide as much as possible
Actually it would be better to hide other crawlers in your htaccess then in your robots.txt ... well... if you're concerned to have a footprint on your pbn...
google can see your robots.txt content, but can't see your htaccess content...
mainceaft
Senior Member
- Apr 10, 2013
- 884
- 185
