ThopHayt
Elite Member
- Jul 25, 2011
- 7,012
- 2,864
Just use the search box in the upper right corner, click into it, click the "more..." button and search block bots. Constrain your search to the Black Hat SEO section and you'll find dozens of guides and threads that walk you through it. This problem has been talked about a lot and there have been detailed solutions posted.
-ThopHayt
-ThopHayt
- Nov 8, 2009
- 8,887
- 9,192
michaelr1988
Regular Member
- Apr 25, 2011
- 489
- 317
In addition to the above, some would argue using .htaccess way is better as it doesn't leave a footprint.
ThopHayt
Elite Member
- Jul 25, 2011
- 7,012
- 2,864
Lets be honest, both leave a footprint. I'm sure Google can figure out who is blocking bots that claim to be Moz in their .htaccess as well. Which is why honestly if you're building large scale you need to mix up how/which/if you block various bots.
-ThopHayt
-ThopHayt
andrew1978
Regular Member
- May 13, 2012
- 440
- 32
Use Spider Spanker
if you dont want bots to crawl your site
if you dont want bots to crawl your site
pelangi
Regular Member
- Jul 30, 2016
- 218
- 37
thats right, you can make some diversify/mixing to it.
or, just leave them...
i just heard about it, how it works?