[Method] How to stop competitors spying on your links

strongvpn · May 14, 2014

I hope it will help me to hide from grassers,Useful, thank you!

royserpa · May 14, 2014

randybishop said:
Has anyone tested?
Does it work?

Yes it works pal

leobar · May 14, 2014

Thanks for sharing OP.
this method works, but its not 100% efficient and may cause you troubles on indexing. i will explain you why.

1. Google does not only use Google Bot, so if you are disallowing everything except Google, you will not get indexed.

2. most people use ahrefs , semrush and seomoz (i will not bother adding all machines you will have to check what else is used but those 3 are the main ones). by blocking those ones you can discourage the lazy competitors but not everyone. if i were your competitor i would just: Google.com/ncr and then type Youtsite.tld open an excel and write down all links indexed by google pointing at your site.
it will not show 100% of your backlinks, but it will show the most powerfull ones (those which your competitors care anyway)

i would say with those robots below you would be fine to stop the lazy competitors from spying on you:

User-agent: Rogerbot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /

I Agree 100% that robots should be added before implementing backlinks or otherwie they stay on those robots history records.

Best Wishes and thanx for sharing.

Leobar!

Edit: point 2 i used before i started using software on inspecting competitors. its kind of old fashion but still works if you have the time

srabantipaul1983 · May 14, 2014

This rules can also be Implement on the Blog Networks for Stop Spying...

TheVigilante · May 14, 2014

I think there is also a plugin that does a similar job but you need a WP website

V · Jun 8, 2014

I will try it and if I run into any problems, I'll post it here.

Thanks for the share OP and leobar

Flameclaw · Jun 8, 2014

Kind of feels like overkill. But thanks.

AlejandroSu · Jun 8, 2014

it is working yet?

sudorank · Jun 8, 2014

abhi007 said:
I think there is also a plugin that does a similar job but you need a WP website

Ye but this method is easy to put together yourself, it's only robots.txt and .htaccess files - no biggie.

West Sider · Jun 8, 2014

nice sharing. liked it.

randybishop · Jun 9, 2014

abhi007 said:
I think there is also a plugin that does a similar job but you need a WP website

Here it is: Free plugin for WP

mic88 · Jun 9, 2014

robots.txt is a file which is readable for everyone. It is better not use it becouse it is a footprint.

mainceaft · Jun 9, 2014

from experience MJ12bot changed their user agent and acting like regular user
best way to ban them is banning all their hosting IP range use http://myip.ms/ for that
and add Owner CIDR to your .htaccess file like this
178.63.0.0/16
I hate all those snaky bots they cost my a lot because competitions reports .

ohfoshodawg · Jun 9, 2014

Great info, thanks.

ezines · Jul 23, 2014

mainceaft said:
from experience MJ12bot changed their user agent and acting like regular user
best way to ban them is banning all their hosting IP range use http://myip.ms/ for that
and add Owner CIDR to your .htaccess file like this
178.63.0.0/16
I hate all those snaky bots they cost my a lot because competitions reports .

Yes, looks like their crawler cannot be blocked totally. They've mentioned it this blogpost;

Code:

http://blog.majesticseo.com/general/verifying-site-majesticseo-every-seo-audit/

But, how to determine the block of IP addresses they're using?

sudorank said:
Ye but this method is easy to put together yourself, it's only robots.txt and .htaccess files - no biggie.

Also, for those who have installed iThemes Security in WordPress can just add the crawlers in block user agent section in the settings area, or if Yoast SEO is installed it is also easy to add the crawlers into its robots.txt editor, or directly into the .htaccess file editor.

worldatlarge · Nov 3, 2014

Is there an automated way to block these backlink crawler bots that keeps an up to date list of their IPs and User Agents?

It's one thing to block user agents like:
ahrefsbot
mj12bot
rogerbot
SemrushBot

But what if they use different user agents? What if they keep changing their IP addresses? If you are a webmaster with dozens or hundreds of sites, it is nearly impossible to try to keep up with all the changes. Especially if you use different CMS platforms.

BlackHatRael · Nov 11, 2014

Found this:

"How exactly do you keep blocking ahrefs, majecticseo etc.. ?

You have to install and use this plugin on all your PBN sites: http://spyderspanker.com/" from:

HTML:

http://www.blackhatworld.com/blackhat-seo/black-hat-seo/697565-pbn-q-ask-me-anything-about-how-i-build-manage-rank-pbns.html

Devil Rider · Apr 4, 2016

Thanks for sharing buddy...nice post

TheVigilante · Apr 4, 2016

Devil Rider said:
Thanks for sharing buddy...nice post

You bumped a two year old thread just to say thanks, when there is a thanks button for that?

Idiot!

Furious Man · Apr 4, 2016

I just learned something today!

Thanks for taking the time to explain that

[Method] How to stop competitors spying on your links

Newbie

BANNED

Regular Member

Newbie

Banned - Multiple Rules Violations

Elite Member

Junior Member

Junior Member

Banned - Selling via PM

Newbie

Elite Member

Registered Member

Senior Member

Newbie

Power Member

Regular Member

Junior Member

BANNED FOR SPAM/FAKE REVIEWS

Banned - Multiple Rules Violations

Elite Member

Main Menu

Marketplace

Making Money

BlackHat World