Block Semrush,ahref,moz

radichone

Junior Member
Joined
Nov 8, 2015
Messages
119
Reaction score
5
Hello
Can you tell me which code should i use please to block all web crawlers and only allow search engine to find my site.
I have already found some answers here but most of it were old.
Is it a good idea only to allow search engine to crawl my site and block all the rest?
 
Well if you are serious about your website and might use these tools in future to track your/competitor's SEO performance, then I won't recommend blocking these sites using robots.txt
what is the reason behind blocking these sites? If you are worried that your competitor might check your backlinks or something like that then also blocking these crawlers will hardly do any good.
 
You need a cloaker or other php script that detects these crawlers and block them.
 
If you wan't to block them, simply modify your robots.txt file. However, not every backlink tool respects robots.txt.

Buuuut: you can't block domains linking to your domain. You can just prevent your domain to be crawled == outgoing links from your domain won't be visilbe, but yes incoming links. This in turn means: the backlinks to your domain will be still visible! Not suited to stop potential competitors to see your backlinks.
 
I don't recommend blocking sites like ahrefs, moz because then you don't get to track your competitor's performance, how are they getting backlinks....
 
Back
Top