Hello All, I want to send over 7 million get request to a website api in 1hour. How can I achieve it? After sending couple of thousand request the target website automatically closes my connection. Anyone with any ideas?
If you send 350k on 20 devices its still 7 Million or 70 x 100k Find a number you doesnt get blocked and play with It
Any recommendation for cheap proxy pool services providing above 5k proxies which can be used concurrently? Couldn’t find any which can do this.
From my limited testing.. 5k probably. The main problem is I need to do this in an 1hr, so the amount of request that each device need to send in 1sec is over quite high.
Even if you use proxies, if the scraper is not done right, and leaves footprints, they can still block you. Apart from using proxies, make the scraper so each request is unique.
That sounds like a DDOS - maybe they have setup a protection in place for their API for these kind of actions.
First to know, what's the device fingerprinting, such as User-agent, Canvas... (IP address is important to factor also). To Send requests without got block... check robots.txt at first, Then check offer API or not, To avoid device fingerprinting issue, you can use Headless browsers, such as Selenium and Puppeteer I write a post about, How to prevent getting blacklisted or blocked when scraping I think you can learn it, though I do not know your purpose to sent requests, I think many are similar to scraper.
Sorry, no experience, I think no matter use what, too many requests from one IP/server are impossible...
