Amazon scraper + ghosted Ip?

Hey there,

General idea: Have an ebay to amazon scraper that finds Amazon items and pulls ASIN #s into a text file.

Everything started out fine, i ran the bot at about 200 items and it pulls a bunch of ASIN's... but once I bumped the scraper up to 1k items it came up with an error and now will not pull ANY ASIN's. It runs through all items within a minute without pulling anything.. whereas before it would take 5 to 10 minutes

Is it possible that amazon ghosted my ip? If so, why and what is the best course of action to take

Also, I would really love to be able to use this thing on a mass scale. Is there any way to give this bot the ability to scrape thousands of items (on both ebay and amazon) without running into a problem?

Out of my realm of understanding so any help/advice is seriously appreciated

-Zach

How are you scraping atm?

By using the official APIs it should be easy to get what you want.

Afaik Ebay has a max call limit per 24h, which starts at 5000 calls.
That limit could be raised if you get your app approved (afaik).

Amazon had some limits that were only x calls/second (60seconds). It's somewhere in their tos.

Yep they have blocks on IPs after X amount of requests over Y time. Get some rotating proxies

jamie3000 said:

Yep they have blocks on IPs after X amount of requests over Y time. Get some rotating proxies

Click to expand...

A spammer bumped this necro thread mate, so I'll lock this.