How many connections for public proxies in Scrapebox?

On a 100mbit VPS what is the max connections I should set for URL harvesting on scraped public proxies? They seem to burn out within about 10 minutes if I go much above 100 in the max connections. Should I just try to add as many proxies as I have connections, or will I really need to lower the connections to stop them burning out so fast? What about timeout settings?

Some people recommend to buy private proxies and set lower max connections (e.g., 5) and timeout values to stop burn out. Will private proxies still scrape as fast on lower max connections?

Eventually I want to start using a tool like yaph to portscan for my own proxies so I can just keep adding them in bulk without caring about burn-out, but it looks like it will take a lot of trial and error to go through all the known country cidrs and find some good ranges. That said, I just want to make do with public proxies for the time being.

They are public. So there is you and maybe 200 other people using them.

Whatever setting you make will on average make 1/2 a percent difference to the overall workload those proxies are doing. (1 in 200 = 1/2 a percent)
I would use them as quickly as you can. If you don't burn them out - the other 199 people using them certainly will.
Might as well get as much juice out of them as you can before that happens.

Private proxies will scrape faster even on less connections. Especially if you count the time saved when public ones burnout then you have to scrape and test more. Private is the way to go even if you only have 5 connections running (you will need 30+ to do this).

i got a bann from that

rubyakerman said:

i got a bann from that

Click to expand...

On your proxies? Then you were probably using too many connections. If your running private you don't need many to scrape fast. Of course having more proxies and less connections will keep you from getting banned.

These days I use 10 proxies minimum for ONE THREAD.
With Yahoo for some reason it's 20 per thread to avoid 999 errors.

3 years ago those ratios were the other way around.

Scritty