Wilson Grant Fisk
Elite Member
- Nov 10, 2012
- 15,677
- 46,839
Uhm, if you read what I wrote you already saw that I have no problem blocking all crawling bots with the robots.txt and .htaccess file, in fact 150+ bots.
What you wrote was :
"In fact, no matter how you try (robots.txt, htaccess, etc) they will often still ignore it."
NOPE they will NEVER ignore it when you set it up properly.
I am telling you that now matter what you have done/do there are still bots that WILL ignore your robots.txt
Blocking 150+ bots is nothing, as I already stated, they are very easy to code.