1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Block spider "Jakarta Commons-HttpClient/3.0" that belongs to VeriSign?

Discussion in 'Black Hat SEO' started by Shameless, Apr 17, 2008.

  1. Shameless

    Shameless Newbie

    Joined:
    Mar 17, 2008
    Messages:
    41
    Likes Received:
    3
    Home Page:
    Yesterday I implemented a new method to my site vids2watch.com wich resulted into huge spider activities over night (12h).

    The most crawling spider is "Jakarta Commons-HttpClient/3.0" from IP 69.36.158.7 (hxxp://who.is/whois-ip/ip-address/69.36.158.7/) wich belongs to VeriSign. In numbers, this spider crawled over 100,000 times in 12h.

    So, why do they crawling like crazy (~70% more than others)? Why VeriSign? And if it doesnt belong to search engines, should I block it to save bandwidth (it eat up 350mb so far).

    Glad to hear your opinions.
    Cheers.
     
  2. tukhat

    tukhat Registered Member

    Joined:
    Feb 8, 2008
    Messages:
    85
    Likes Received:
    57
    Location:
    Everyday is Holiday
    Jakarta? does it mean it comes from indonesia?