1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Using Amazon EC2 as a proxy?

Discussion in 'Proxies' started by soapradio, Aug 9, 2011.

  1. soapradio

    soapradio Newbie

    Joined:
    Aug 9, 2011
    Messages:
    2
    Likes Received:
    0
    I'm considering using Amazon EC2 as a proxy to scrape a website's live data through PHP/cURL. The website throttles based on IP. There are thousands of requests involved.. The requests need to be made live (as users search on my site) because the data changes throughout the day.

    Is Amazon EC2 a good choice for a proxy here? Will scraping on their service draw Amazon's ire?

    Note: This site isn't scraped by many people, so they're definitely not setup to block Amazon IP's specifically.
     
  2. napoleon007

    napoleon007 Newbie

    Joined:
    Feb 17, 2011
    Messages:
    4
    Likes Received:
    2
    I use ec2 regularly as a proxy for scraping google (keyword research). It actually works really well. I haven't had any complaints from amazon.

    At times it can take up to a minute to spin up an instance, so you would want to have a background job to do that. The tiny instances work really well as squid proxies and are cheap.

    Also, if you go with squid be sure to use the latest version, since older versions aren't completely transparent.
     
    • Thanks Thanks x 1
  3. soapradio

    soapradio Newbie

    Joined:
    Aug 9, 2011
    Messages:
    2
    Likes Received:
    0
    Thanks napoleon007.Will check this out
     
    Last edited: Aug 9, 2011