There is no point in using 500 threads with that few number of public proxies. If I read your OP correctly, you're using 415 public proxies.
I GUARANTEE you, that of those 415 public proxies, not all of them are going to be working & certainly only about one tenth of them are going to have a latency of under 1000ms.
Lets say that you're incredibly lucky and 400 of those 415 public proxies are working. Now lets say you're SUPER incredibly lucky and those 400 proxies have a latency of 3000ms (that latency is tested using a 1kb packet).
First, lets take a look at connections:
500 connections using 400 public proxies (if all are working...which I can assure you they aren't). That means that 100 of those public proxies are being forced to take two connections at the same time.
Now, latency. Using a 1kb packet, your proxies have a latency of 3000ms. Now you're trying to gather ALLLLLLL this data...you're REALLY putting a strain on these proxies...imagine how many other people are using them.
What's likely happening, is SB is spending more time looking for a viable proxy to use than it is actually scraping URLs.
I can scrape at 1,500 URLs a second on my dedi using less proxies & less connections....less than a fifth of what you're using....know how?
By using private proxies....and lots of them.
Now, if you don't want to have to put out that kind of cash but want to scrape at much higher speeds, you need make sure you're using only the best of the best public proxies.
You do this by repeat filtering...lots of repeat filtering.
Go to SB's proxy tester, remove duplicates and test your proxies. Remove the dead ones. Now, remove proxies that have a latency of over 10 seconds (10,000ms).
Test them again. Remove dead ones. Remove latency of over 10 seconds.
Test them again. Remove dead ones. Now remove latency of over 5 seconds.
Test them again. Remove dead ones. Remove latency of over 2 seconds.
Test them again. Remove dead ones.
Test them again. Remove dead ones.
Do this until you get ZERO dead proxies (it will happen....eventually).
Now, remove proxies that have a latency of over 2000ms.
If you're only getting your public proxies from SB, than you're going to have about 50 proxies left after all of this.
Using 15 connections and those 50 proxies, you'll harvest at almost 300 URLs per second.
Twice the speed as what you're currently getting & with only a quarter of what you're settings are currently at.
You can't make a chevy cavalier perform like a ferrari, no matter how hard you push on the gas pedal.
You also can't make SB perform faster & better just by cranking up the settings. You have to make sure you're giving it the best "parts" before you even think about laying on the "gas".
