ANybody do web scraping at large scale?

butterflygarden

Newbie

Mar 23, 2022

13

3

• Wednesday at 10:19 PM

• #1

I'm just wondering how you people do it since the same IP address tends to get blocked quite a lot.
Do you do IP rotation, slow down the process, or use residential proxies? What has worked for you guyss?

 

app_pilot

Senior Member

Jr. VIP

Jun 21, 2024

843

202

• Wednesday at 10:58 PM

• #2

Slow your requests way down and rotate residential proxies

 

IPcook_Proxy

Registered Member

Marketplace seller

Jul 14, 2026

58

6

• Yesterday at 4:07 AM

• #3

Most of our users rely on rotating residential proxies to avoid blocks on stricter sites. For platforms with lower security, they usually opt for datacenter proxies to maximize speed and cost efficiency. Slowing down request rates helps, but using rotating IPs makes the real difference.

 

maks23092004

Newbie

May 6, 2024

30

3

• Yesterday at 9:20 AM

• #4

I know a few people who order around 50–100 mobile proxies for these purposes. They’ve told me that it has become noticeably harder recently, though, so even with mobile proxies it’s not as straightforward as it used to be.

 

E

extremeboy

Elite Member

Jr. VIP

Jul 8, 2010

4,008

824

• Yesterday at 11:45 AM

• #5

what website you want to do scraping? Maybe i can help and join you use as in experiment if it's appropriate for me.

I had done a lot of it in the past time. Millions of pages monthly at scale.