1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Amazon scraper + ghosted Ip?

Discussion in 'BlackHat Lounge' started by Euphorix, Oct 19, 2015.

  1. Euphorix

    Euphorix Supreme Member

    Joined:
    Feb 20, 2015
    Messages:
    1,331
    Likes Received:
    799
    Location:
    Mountain View, CA
    Hey there,

    General idea: Have an ebay to amazon scraper that finds Amazon items and pulls ASIN #s into a text file.

    Everything started out fine, i ran the bot at about 200 items and it pulls a bunch of ASIN's... but once I bumped the scraper up to 1k items it came up with an error and now will not pull ANY ASIN's. It runs through all items within a minute without pulling anything.. whereas before it would take 5 to 10 minutes

    Is it possible that amazon ghosted my ip? If so, why and what is the best course of action to take

    Also, I would really love to be able to use this thing on a mass scale. Is there any way to give this bot the ability to scrape thousands of items (on both ebay and amazon) without running into a problem?

    Out of my realm of understanding so any help/advice is seriously appreciated

    -Zach
     
    • Thanks Thanks x 1
  2. RightFootFanatic

    RightFootFanatic Regular Member

    Joined:
    May 31, 2015
    Messages:
    347
    Likes Received:
    192
    Occupation:
    DevOps
    Location:
    Whimsyshire
    How are you scraping atm?

    By using the official APIs it should be easy to get what you want.

    Afaik Ebay has a max call limit per 24h, which starts at 5000 calls.
    That limit could be raised if you get your app approved (afaik).

    Amazon had some limits that were only x calls/second (60seconds). It's somewhere in their tos.
     
    • Thanks Thanks x 2