Yesterday I implemented a new method to my site vids2watch.com wich resulted into huge spider activities over night (12h). The most crawling spider is "Jakarta Commons-HttpClient/3.0" from IP 126.96.36.199 (hxxp://who.is/whois-ip/ip-address/188.8.131.52/) wich belongs to VeriSign. In numbers, this spider crawled over 100,000 times in 12h. So, why do they crawling like crazy (~70% more than others)? Why VeriSign? And if it doesnt belong to search engines, should I block it to save bandwidth (it eat up 350mb so far). Glad to hear your opinions. Cheers.