Blocking Majestic and other bots

rayee

Power Member
Joined
Jun 21, 2012
Messages
771
Reaction score
139
I used to had a VA working for me who handled all of my hosts but as the story goes, he left.
Thus, I am now handling all my hosting.

This is my query. I insert the HTAccess in all root folders of the domains but the old hosts that he handled previously wouldn't have it coming up in Majestic even if I were not to put this HTaccess file.

However the new host .. even if I were to put it in the root folder it comes up in Majestic.

Any ideas other than HTaccess what do one do to block the crawler? Is it something to do with the hosting that handles all domains?
 
You can deny through robots.txt but it's a footprint and might look PBNish Your best bet is to block via htaccess (or httpd main config) on known IP address ranges for unwanted crawlers. I wouldn't even block on the user agent or anything else that's spoofable. Yes I'm paranoid and I don't even use PBNs!
 
Back
Top