1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Amazon Scraping / Proxies?

Discussion in 'General Programming Chat' started by Steve Books, Jun 21, 2016.

Tags:
  1. Steve Books

    Steve Books Newbie

    Joined:
    Jun 14, 2016
    Messages:
    2
    Likes Received:
    1
    Hi
    I wrote a program that searched a list of items (300K-500K) on amazon for price changes. The needs to be fast. The faster I search the more I get IP bans. Anyone here have experience with this and can give me some advice - like whether to use proxies, what kind of proxies, how many, from where to get them?
    Thanks in advance.
    Steve
     
    • Thanks Thanks x 1
  2. NiceMicha

    NiceMicha Newbie

    Joined:
    Aug 5, 2016
    Messages:
    16
    Likes Received:
    0
    Gender:
    Male
    Hello,
    i am searching for the same advice...
    is there anyone with experiance in scraping amazon Items?
    @Steve Books : i am interested in Items with Reviews only, how do you save them?
    Maybe we can help each other..
     
  3. Pecola

    Pecola Newbie

    Joined:
    Aug 23, 2016
    Messages:
    11
    Likes Received:
    0
    Gender:
    Male
    Same issue with me, although I'm getting CAPTCHAs thrown back at me. I'm using the TOR network to anonymize my scraping.
     
  4. Steve Books

    Steve Books Newbie

    Joined:
    Jun 14, 2016
    Messages:
    2
    Likes Received:
    1
    Hi
    I would love to discuss this topic w/you, I have been working with this for over a year and I'm sure we can help each other. send me info how to contact dica1101 at gmail
     
  5. Pecola

    Pecola Newbie

    Joined:
    Aug 23, 2016
    Messages:
    11
    Likes Received:
    0
    Gender:
    Male
    Hey I just sent you an email. Let me know if you received it.
     
  6. azurous

    azurous Jr. VIP Jr. VIP

    Joined:
    Jan 30, 2015
    Messages:
    339
    Likes Received:
    70
    Gender:
    Male
    Occupation:
    Fat Cat
    Location:
    New York
    Home Page:
    Is there a reason you can't use the amazon APIs instead of scraping?
     
  7. Pecola

    Pecola Newbie

    Joined:
    Aug 23, 2016
    Messages:
    11
    Likes Received:
    0
    Gender:
    Male
    For myself personally, I'm looking to scrape the Best Sellers list, as I don't have a list of ASINs available. What I remember from my research a few months ago is that doing something like this isn't possible using the Amazon API.

    Also, I believe there is a daily cap on the number of requests you can make using the Amazon API, although that might be changed now.
     
  8. azurous

    azurous Jr. VIP Jr. VIP

    Joined:
    Jan 30, 2015
    Messages:
    339
    Likes Received:
    70
    Gender:
    Male
    Occupation:
    Fat Cat
    Location:
    New York
    Home Page:
    I have some experience scraping with proxies and I advise you to get private, fresh ones (buyproxies.org maybe?) otherwise it'll be slow. Anyway, use a proxy, if you get banned, cut your scrape rate by half, try again. Repeat with another proxy.
     
  9. Pecola

    Pecola Newbie

    Joined:
    Aug 23, 2016
    Messages:
    11
    Likes Received:
    0
    Gender:
    Male
    Precisely what I'm currently trying. I'm scraping at a rate of one request per 30 seconds, which I feel is relatively very slow. However, I'm still getting CAPTCHAs being sent my way after about 30 minutes of scarping, which indicates to me that Amazon has flagged my proxies as possible bots.
     
  10. azurous

    azurous Jr. VIP Jr. VIP

    Joined:
    Jan 30, 2015
    Messages:
    339
    Likes Received:
    70
    Gender:
    Male
    Occupation:
    Fat Cat
    Location:
    New York
    Home Page:
    Where are you proxies from?
     
  11. Pecola

    Pecola Newbie

    Joined:
    Aug 23, 2016
    Messages:
    11
    Likes Received:
    0
    Gender:
    Male
    Initially I bought a set of 10 proxies from ActProxy.
     
  12. azurous

    azurous Jr. VIP Jr. VIP

    Joined:
    Jan 30, 2015
    Messages:
    339
    Likes Received:
    70
    Gender:
    Male
    Occupation:
    Fat Cat
    Location:
    New York
    Home Page:
    I meant where are the proxies located. US? are they private?
     
  13. Pecola

    Pecola Newbie

    Joined:
    Aug 23, 2016
    Messages:
    11
    Likes Received:
    0
    Gender:
    Male
    Ah yes, they were all US west proxies, and they were all private.