PBN: Is blocking spiders with htaccess a footprint?

[aussiejack]

I built some PBN sites and want to block majestic/ahrefs spider etc via the htaccess file now. However, I am not sure if the google bot sees that and if it is a big footprint if I do this for all PBN sites?

 

[iamsolo]

You're so much paranoid about google. I dont think that was an fp. Why are you blocking bots in pbn sites? Blocking them in money site is enough.

 

[crazedspyker]

You're so much paranoid about google. I dont think that was an fp. Why are you blocking bots in pbn sites? Blocking them in money site is enough.

Because he doesn't want his competitors to easily find his entire PBN network. Blocking it on money site doesn't really make sense in this regard.

And OP, I don't think it's a footprint like robots.txt and spyderspanker. You should be fine, a lot of legit sites block these robots for other purposes. If you are still paranoid, then just vary it up. Use .htaccess, spyderspanker, robots.txt at random. I don't bother with that personally.

 

[aussiejack]

ok thanks man - -

 

[tb303]

.htaccess is server side stuff.

unless google has root access to your server they aint seeing it!

 

[Leith]

I do that too, it should be fine.

 

[rogerke]

.htaccess is server side stuff.

unless google has root access to your server they aint seeing it!

They can just fetch under a different useragent than googlebot (for example MJ12bot from Majestic or ahrefsbot from ahrefs) and notice you're blocking these bots.

Wether it's a footprint depends on wether they will do this and take action against sites like that. One of the complicated factors is mentioned earlier in this thread; a lot of legitimate sites block these bots as well, so I don't think they will ever take algorithmic actions against these sites. They might be passed on for manual review however. Nobody outside Google can give a valuable answer about this.

One way to prevent it is by blocking IP ranges (of those backlink blocks) instead of useragents, but that's a bit tricky because you need to do a lot of due dillegence and the IP ranges might change overnight.

All in all it's definitely worth it to block these bots in .htaccess. Not blocking them is simply not an option because you might as well paint a huge target on your head for your competitors to shoot at.

 

[Expertpeon]

They can just fetch under a different useragent than googlebot (for example MJ12bot from Majestic or ahrefsbot from ahrefs) and notice you're blocking these bots.

Google routinely testing useragents they don't control is likely illegal (or at the very least, a nice civil suit for anyone they impersonate, though it could indeed be illegal to do so). Furthermore, server logs would tell you if Google did this, pretty massive risk for little reward.

 

[rogerke]

Google routinely testing useragents they don't control is likely illegal (or at the very least, a nice civil suit for anyone they impersonate, though it could indeed be illegal to do so). Furthermore, server logs would tell you if Google did this, pretty massive risk for little reward.

Interesting. Any source to do some further reading? I'm not familiar with US law, but would love to read why this could be illegal. Pretty sure Google already uses different useragents than googlebot to catch cloaking etc., so I'm pretty sceptical.

If you're right blocking these bots in .htaccess is even safer than I thought though.

 

[gazmo]

Thanks, OP, now you made me paranoid too.

In all seriousness though, I've never had a problem with that. Hopefully this won't be considered a footprint in the future either (although you're right that it can raise suspicion).

 

[Expertpeon]

Interesting. Any source to do some further reading? I'm not familiar with US law, but would love to read why this could be illegal. Pretty sure Google already uses different useragents than googlebot to catch cloaking etc., so I'm pretty sceptical.

If you're right blocking these bots in .htaccess is even safer than I thought though.

Manipulating user-agents in order to bypass blocked content from crawlers or attempts to gain access to sites not permitted by design is typically going to be seen as unlawful in the US. Google does use a few user-agents, but these user-agents are controlled (and likely trademarked) by Google. Now when it's a major company, as Yelp! has proven over and over, they get to play by a completely different set of rules in the US (ie, falsifying reviews and extorting businesses is apparently legal according to the Federal Appeals court).

However, Andrew Auerenheimer did indeed get put in jail in part for spoofing user-agents to gain access (or "test" the services) to a website. This is all the guy did, and it was seen as a hacking attempt. If Google did this, (granted it's a stupid idea to do so on their part due to the lack of information it would provide) I imagine it'd be lawsuit material.

https://www.eff.org/files/2014/04/11/weev.pdf

 

[the_demon]

The main reason Google is unlikely to test .htaccess blocking is because they would need to use a lot more IPs to avoid being detected and also it would take a lot more computational resources as you would need to check over and over again using various user agents. Also, some services like CloudFlare and Distil Networks might block these bots for unusual behavior if they did this.

 

[ThopHayt]

Doing any one behavior across an entire network is probably not a great idea. As others have said I have no evidence Google checks htaccess but I cannot say for sure they don't. Nor can I say it is unlikely they will not eventually.

My opinion: Mix it up, don't "always" do it.

 

[xha44a]

Hi OP

Google has no way of accessing your htaccess. You can't view it via your web browser. Go ahead and try. It's not something Google can access. It merely tells Apache what to do with certain visitors/bots etc. I wouldn't worry about it.

 

[Laubster]

Lot of people have asked this, but there are legitimate reasons for blocking bots outside of SEO, mainly to reduce server load. At this point its not a footprint, and most guys I know with insane PBN's are blocking every bot imaginable. If they're doing it with their 5 and 6 figure networks I think it's ok.