Amazon Scraping / Proxies?

[Steve Books]

Hi
I wrote a program that searched a list of items (300K-500K) on amazon for price changes. The needs to be fast. The faster I search the more I get IP bans. Anyone here have experience with this and can give me some advice - like whether to use proxies, what kind of proxies, how many, from where to get them?
Thanks in advance.
Steve

 

[NiceMicha]

Hello,
i am searching for the same advice...
is there anyone with experiance in scraping amazon Items?
@Steve Books : i am interested in Items with Reviews only, how do you save them?
Maybe we can help each other..

 

[Pecola]

Same issue with me, although I'm getting CAPTCHAs thrown back at me. I'm using the TOR network to anonymize my scraping.

 

[Steve Books]

Same issue with me, although I'm getting CAPTCHAs thrown back at me. I'm using the TOR network to anonymize my scraping.

Hello,
i am searching for the same advice...
is there anyone with experiance in scraping amazon Items?
@Steve Books : i am interested in Items with Reviews only, how do you save them?
Maybe we can help each other..

Hi
I would love to discuss this topic w/you, I have been working with this for over a year and I'm sure we can help each other. send me info how to contact dica1101 at gmail

 

[Pecola]

Hi
I would love to discuss this topic w/you, I have been working with this for over a year and I'm sure we can help each other. send me info how to contact dica1101 at gmail

Hey I just sent you an email. Let me know if you received it.

 

[azurous]

Is there a reason you can't use the amazon APIs instead of scraping?

 

[Pecola]

Is there a reason you can't use the amazon APIs instead of scraping?

For myself personally, I'm looking to scrape the Best Sellers list, as I don't have a list of ASINs available. What I remember from my research a few months ago is that doing something like this isn't possible using the Amazon API.

Also, I believe there is a daily cap on the number of requests you can make using the Amazon API, although that might be changed now.

 

[azurous]

I have some experience scraping with proxies and I advise you to get private, fresh ones (buyproxies.org maybe?) otherwise it'll be slow. Anyway, use a proxy, if you get banned, cut your scrape rate by half, try again. Repeat with another proxy.

 

[Pecola]

I have some experience scraping with proxies and I advise you to get private, fresh ones (buyproxies.org maybe?) otherwise it'll be slow. Anyway, use a proxy, if you get banned, cut your scrape rate by half, try again. Repeat with another proxy.

Precisely what I'm currently trying. I'm scraping at a rate of one request per 30 seconds, which I feel is relatively very slow. However, I'm still getting CAPTCHAs being sent my way after about 30 minutes of scarping, which indicates to me that Amazon has flagged my proxies as possible bots.

 

[azurous]

Where are you proxies from?

 

[Pecola]

Where are you proxies from?

Initially I bought a set of 10 proxies from ActProxy.

 

[azurous]

I meant where are the proxies located. US? are they private?

 

[Pecola]

I meant where are the proxies located. US? are they private?

Ah yes, they were all US west proxies, and they were all private.