1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scraper with thousands of IP's needed

Discussion in 'Hire a Freelancer' started by scrapy, Sep 14, 2010.

Tags:
  1. scrapy

    scrapy Newbie

    Joined:
    Sep 7, 2010
    Messages:
    1
    Likes Received:
    0
    I have a list of close to 100 million URL's (about 500gb of data) from one site I need downloaded. I have tried using some of the proxy services available but cannot get enough IP's at a low enough price to scrape slow enough so that the IP's do not get blocked.

    As a minimum I expect you will need 5,000 IP's and no more than 1 simultaneous connection per IP.

    We have all the code written and the list of URL's so all I need is someone with a large network of IP's to run it. If you can provide the IP's as a normal proxy service we can run it from our own server. Or you can run the script on your own server and just provide me with the finished data dump.

    If you are keen on performing this service please provide a quote. I can pay via LR, PP, AP, or direct wire transfer into your account.
     
  2. guptainfosys

    guptainfosys Jr. VIP Jr. VIP Premium Member

    Joined:
    Jun 30, 2010
    Messages:
    1,739
    Likes Received:
    292
    Occupation:
    SEO Developer and Internet Marketer
    Location:
    Bharatpur, India
    Home Page:
  3. portalweb

    portalweb Supreme Member Premium Member

    Joined:
    Jan 7, 2010
    Messages:
    1,415
    Likes Received:
    547
    Occupation:
    Hard Core Engineer
    Location:
    New York
    My suggestion:

    Why don't you spread out the scraping script to be scheduled over a period of time with a set of proxies behind your script, instead of "one-shot" huge proxies offered by proxies providers? Remember, the proxies die easily.

    If you are interested in technical assistance on your scraping needs, please feel free to PM me.