1. This website uses cookies to improve service and provide a tailored user experience. By using this site, you agree to this use. See our Cookie Policy.
    Dismiss Notice

Sending millions of get request

Discussion in 'Programming' started by nihalar, Nov 7, 2019.

  1. nihalar

    nihalar Junior Member

    Joined:
    Apr 18, 2012
    Messages:
    141
    Likes Received:
    13
    Location:
    Qatar
    Hello All,

    I want to send over 7 million get request to a website api in 1hour. How can I achieve it? After sending couple of thousand request the target website automatically closes my connection.
    Anyone with any ideas?
     
  2. AngelSeo

    AngelSeo Jr. VIP Jr. VIP

    Joined:
    Nov 3, 2017
    Messages:
    1,678
    Likes Received:
    954
    Gender:
    Male
    Proxies, lots of them?
     
    • Thanks Thanks x 1
  3. liyong601c

    liyong601c Jr. VIP Jr. VIP

    Joined:
    Mar 30, 2010
    Messages:
    368
    Likes Received:
    77
    Occupation:
    SEO,PPV,PPC
    Location:
    Mars
    Home Page:
    That's why there are lots of residential proxies online!
    Use it never got blocked...
     
  4. BrazilianBusinessman

    BrazilianBusinessman Jr. VIP Jr. VIP

    Joined:
    Mar 18, 2019
    Messages:
    678
    Likes Received:
    334
    Gender:
    Male
    Occupation:
    Detective of my own failure
    Location:
    The place where humans aint slaved by money
    If you send 350k on 20 devices its still 7 Million or 70 x 100k
    Find a number you doesnt get blocked and play with It
     
  5. nihalar

    nihalar Junior Member

    Joined:
    Apr 18, 2012
    Messages:
    141
    Likes Received:
    13
    Location:
    Qatar
    Any recommendation for cheap proxy pool services providing above 5k proxies which can be used concurrently?

    Couldn’t find any which can do this.
     
  6. nihalar

    nihalar Junior Member

    Joined:
    Apr 18, 2012
    Messages:
    141
    Likes Received:
    13
    Location:
    Qatar
    From my limited testing.. 5k probably. The main problem is I need to do this in an 1hr, so the amount of request that each device need to send in 1sec is over quite high.
     
  7. Celil Yaman

    Celil Yaman Junior Member

    Joined:
    Feb 24, 2019
    Messages:
    101
    Likes Received:
    14
    Gender:
    Male
    hmmmm btw
     
    • Thanks Thanks x 1
  8. bagle

    bagle Newbie

    Joined:
    Nov 7, 2019
    Messages:
    4
    Likes Received:
    0
    Gender:
    Male
    You will get blocked when you fire 5K requests per 1hour, is this correct?
     
  9. nihalar

    nihalar Junior Member

    Joined:
    Apr 18, 2012
    Messages:
    141
    Likes Received:
    13
    Location:
    Qatar
    Ys its correct
     
  10. TomTheCat

    TomTheCat Junior Member

    Joined:
    Oct 27, 2019
    Messages:
    138
    Likes Received:
    54
    Gender:
    Male
    Even if you use proxies, if the scraper is not done right, and leaves footprints, they can still block you. Apart from using proxies, make the scraper so each request is unique.
     
  11. virtualpurity

    virtualpurity Jr. VIP Jr. VIP

    Joined:
    Nov 12, 2012
    Messages:
    1,221
    Likes Received:
    732
    Occupation:
    SEO, Hosting
    Location:
    /root
    Home Page:
    That sounds like a DDOS - maybe they have setup a protection in place for their API for these kind of actions.
     
  12. turelink

    turelink Junior Member Premium Member

    Joined:
    Jul 26, 2015
    Messages:
    179
    Likes Received:
    35
    First to know, what's the device fingerprinting,
    such as User-agent, Canvas... (IP address is important to factor also).

    To Send requests without got block...
    check robots.txt at first,
    Then check offer API or not,
    To avoid device fingerprinting issue, you can use Headless browsers, such as Selenium and Puppeteer
    I write a post about,
    I think you can learn it, though I do not know your purpose to sent requests, I think many are similar to scraper.
     
  13. nihalar

    nihalar Junior Member

    Joined:
    Apr 18, 2012
    Messages:
    141
    Likes Received:
    13
    Location:
    Qatar
    Thx.
    I was wondering if i could do this in aws lambda without any proxies?
    Any experience with it?
     
  14. turelink

    turelink Junior Member Premium Member

    Joined:
    Jul 26, 2015
    Messages:
    179
    Likes Received:
    35
    Sorry, no experience, I think no matter use what, too many requests from one IP/server are impossible...