how many proxies do I need?
- Thread starter ojsun
- Start date
If you're scraping simple websites, start with 1-5 rotating proxies and add more if needed. For stricter sites, use 20+ proxies or a big rotating pool. Why not go for a good mobile proxy since they can work alone since mobile IPs are harder to block?
- Feb 28, 2010
- 5,501
- 4,420
Yeah just start with a couple proxies and if you need more just add them until you find the perfect amount
nickjohn528
Junior Member
- Jan 22, 2025
- 103
- 39
Mate only one proxy enough
ristoriel
Banned - Multiple Rules Violations
- Aug 24, 2020
- 4,184
- 1,658
For 10 threads, you’ll generally need at least 10-20 rotating residential proxies to avoid getting blocked, but it depends on the site’s anti-bot measures. If the proxies rotate with every request, you might get away with fewer.
matawalleyayikokari
Junior Member
- Sep 19, 2024
- 131
- 54
As a rough estimate, you may need 100-500 agents, depending on the replacement frequency and the length of the crawl. If you run the crawler for a long time, it is recommended to start with 500 agents to ensure stability.
D
Deleted member 731332
Guest
If you’re scraping google and you want all the results you can get, for 10 threads you need 100 proxies. Even if they are rotating. Many other people use them. I personally use dedicated proxies at the same ratio. With a 30 second delay.
Public proxies are the best for scraping but you need to be able to scan IP ranges for large amounts of public proxies. Then you don’t need to use delays. But you still need to use a 1:10 ratio of threads to proxies
Public proxies are the best for scraping but you need to be able to scan IP ranges for large amounts of public proxies. Then you don’t need to use delays. But you still need to use a 1:10 ratio of threads to proxies
- Nov 2, 2008
- 59,100
- 26,642
no matter how manh - they willget blocked eventually
- Apr 27, 2016
- 6,349
- 2,538
You need better proxies to scrape Google.
EndlessWorlds
Junior Member
- Nov 4, 2024
- 132
- 31
Best to stick to a 1:1 ratio (one proxy for one account) to stay safe
- Nov 7, 2015
- 746
- 133
If you use one rotating mobile or residential proxy per thread and your tool has the functionality, you can in theory rotate the proxy's IP address every time a thread gets blocked and then continue scraping. This means your threads can operate individually and only rotate when necessary rather than rotate on a fixed schedule.
Hope that helps.
Proxy4Free
Registered Member
- Nov 1, 2024
- 97
- 34
Nowadays, many proxies are available in unlimited quantities. Maybe you need to test how many you need first until you meet your needs.
- Jun 28, 2024
- 587
- 148
I advice you get like 20 proxies or more thereby fetching you more threats
If you use a provider with a rotation for each request, 10 proxies are enough. If you make many requests on a sensitive site, 50 to 100 proxies may be necessary to remain discreet.
ProxyPapa
Regular Member
- Jan 10, 2025
- 201
- 75
There is no definite answer for this. What are you trying to scrape? Start with 5 Rotating Proxies and then scale up
Fishoooo
BANNED
- Mar 13, 2023
- 100
- 18
For scraping with 10 threads at a time, you’ll need at least 10 rotating residential proxies to avoid overloading a single IP and reduce the risk of bans. If you want extra safety, consider using 15-20 proxies to distribute the load further. Test a small batch first to ensure performance.
