Yesterday I implemented a new method to my site vids2watch.com wich resulted into huge spider activities over night (12h). The most crawling spider is "Jakarta Commons-HttpClient/3.0" from IP 22.214.171.124 (hxxp://who.is/whois-ip/ip-address/126.96.36.199/) wich belongs to VeriSign. In numbers, this spider crawled over 100,000 times in 12h. So, why do they crawling like crazy (~70% more than others)? Why VeriSign? And if it doesnt belong to search engines, should I block it to save bandwidth (it eat up 350mb so far). Glad to hear your opinions. Cheers.