1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scraping online shops - which kind of proxies are needed and where to get them?

Discussion in 'Proxies' started by jimbojones123, Jan 9, 2014.

  1. jimbojones123

    jimbojones123 Newbie

    Joined:
    Jan 9, 2014
    Messages:
    3
    Likes Received:
    0
    Hi,

    I am using a web crawler for scraping online shops. I have heard that some online shops block your IP if you are doing too much queries from a single IP address.
    This is where I need HTTP proxies.

    I have no clue what kind of HTTP proxies I need. Anonymous ones? Which country?

    What is a good source for working proxy lists for my purpose? I need API access to that list or a download link because my web crawler will be automated as much as possible.

    Thanks for your help,
    Jim
     
  2. proxygo

    proxygo Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 2, 2008
    Messages:
    13,076
    Likes Received:
    9,145
    Occupation:
    PROVIDING PROXIES FOR GSA SCRAPING.
    Location:
    BHW
    Home Page:
  3. jimbojones123

    jimbojones123 Newbie

    Joined:
    Jan 9, 2014
    Messages:
    3
    Likes Received:
    0
    Thanks for the quick reply.

    Problem is that I don't know what kind of proxies I need. Is the country important? Do I need anonymous proxies?
    Lets say I want to scrape amazon.

    Thanks
     
  4. proxygo

    proxygo Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 2, 2008
    Messages:
    13,076
    Likes Received:
    9,145
    Occupation:
    PROVIDING PROXIES FOR GSA SCRAPING.
    Location:
    BHW
    Home Page:
    the tool your using helps define the type you need
    and which search engines......

    if your scraping google for links you will need google passed proxies 100%
    if your scraping non google search engines, anonymous proxies will do
     
  5. tahworld

    tahworld Regular Member

    Joined:
    Aug 16, 2013
    Messages:
    457
    Likes Received:
    393
    Location:
    ✔✔✔✔✔✔✔
    As far as I know, you can just go ahead and scrape Amazon with any proxy. I could be wrong but I'm pretty sure you can even use public proxies for Amazon.
     
  6. akacash

    akacash Jr. VIP Jr. VIP

    Joined:
    Jan 16, 2010
    Messages:
    836
    Likes Received:
    617
    Location:
    The Beach, USA
    You could probably get away with just about any kind of proxy for scraping as long as they work. If you plan on posting or commenting anything, it gets a little trickier. Most sites don't have the type of query rates as you're talking about. Typically you'll only run into that on something that wants you to signup, so maybe after X amount of requests it'll start to force a sign up or login page.
     
  7. proxygo

    proxygo Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 2, 2008
    Messages:
    13,076
    Likes Received:
    9,145
    Occupation:
    PROVIDING PROXIES FOR GSA SCRAPING.
    Location:
    BHW
    Home Page:
    does the type of proxie all play a part dependent on the scraper
    as there are many scrapers on the market
     
  8. jimbojones123

    jimbojones123 Newbie

    Joined:
    Jan 9, 2014
    Messages:
    3
    Likes Received:
    0
    Thanks for the helpful responses. Now I know that any proxy will work as long as the proxy itsself works. I will try that.

    Anyone knows a good source of working proxies for my purpose? I need an API or something similar so I can automate the proxy usage.
    The proxy-sale section here wasn't very helpful. Either the threads have a bad review or the proxies cannot be retrieved via API.
     
  9. proxygo

    proxygo Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 2, 2008
    Messages:
    13,076
    Likes Received:
    9,145
    Occupation:
    PROVIDING PROXIES FOR GSA SCRAPING.
    Location:
    BHW
    Home Page:
    there are several sellers who sell proxies in ther with api
    you didnt look hard enough
     
  10. extremeboy

    extremeboy Jr. VIP Jr. VIP

    Joined:
    Jul 8, 2010
    Messages:
    3,184
    Likes Received:
    667
    Occupation:
    World Best RANK Tracker SERPCloud.com
    Home Page:
    get private proxies it's best for stable and reliable scraping for any site data to scrape etc ;)
     
  11. artizhay

    artizhay BANNED BANNED

    Joined:
    Nov 21, 2010
    Messages:
    1,867
    Likes Received:
    1,335
    Well some people will tell you to get private proxies, but that's just overkill. What kind of systems are you scraping? For just stores on your basic platforms without high security, any shared proxy would work. They're cheaper, and can be replaced more efficiently than burning a private proxy if you happened to get blocked from a store. Link in the signature will help you out.