10 million url web scrapes per month same website. Need advice.

[notmyrealnameoryear88]

Im trying to figure out the best way to do this. As of right now with a regular IP i get rate limited from visiting pages after 1000 ish. I have all 10 million URLs and my first thought is to use proxies, so i scraped free proxies and rotated them, but that was a waste of time as it immediately got blocked on every single one before it made even one successful request. So I'm curious what advice you guys have (I'm not opposed to paying for a good proxy service at all) and if you have advice on how to make an ip last longer than usual.

 

[ass&tits]

Probably your best bet will be to use some residential proxies and rotate then with your software when your IP gets limited

 

[notmyrealnameoryear88]

Probably your best bet will be to use some residential proxies and rotate then with your software when your IP gets limited

I saw something about using chrome dev tools protocol to bypass cloudflare I know the website im scraping uses cloudflare do you have any knowledge of that?

 

[ass&tits]

I saw something about using chrome dev tools protocol to bypass cloudflare I know the website im scraping uses cloudflare do you have any knowledge of that?

I am not aware of the method using chrome dev tools, but the way to bypass cloudfare depends on the website, so you will have to find out that on your own. A piece of advice that I can give you is to keep an eye on the cf_clearence cookie.

 

[iproxyonline]

My scraping experience is modest, but not exactly zero Here are my tips

• When a site flaggs your current IP (or the current proxy IP), its likely all other requests markers could be flagged too. So, it's smart to watch not just the IP cleanliness, but everything else too (like headless browsers, request headers, avoiding honeypot traps, fingerprinting, stopping repeated failed attempts, etc).
• You can run scraping in multiple threads, each with its own set of IPs + all the other request data. The time saved from running these in parallel can be used to increase the interval between requests from each IP, so its more likely not to be flagged due to too many requests at the time interval.
• And the simplest — use a proxy with free IP rotation. Ofc choose a proxy from a nice provider, not the public