It is rare that I find a product that exceeds my expectations enough to cause me to comment publicly. It is equally rare that I will be so disgusted with a product and its ownership that I will comment publicly.
After having been sorely disappointed with the results and ethics of the owners of a product that sounds like hangout billionaire I purchased a product called hangout domination. This product uses GSA, GScraper and requires lots of proxies. I tried and still do have a subscription to the GScraper proxies but I do not need it. With the GScraper proxies I was getting about 6 million urls a day.
During my first 30 days I experimented with different servers and different settings. There is a part of me that does not want to fully disclose this but here goes. Using the 64gb server(4 cores/8 threads) from OVH split into 4 vps's, scraping with 500 threads and 60 second timeout and running two instances of GScraper on the same vps I have found the following: The memory is barely touched (we have 16gb) during the scrape but is not enough to load even a fourth of the proxies scraped in a day, dedupe them and export (usually the blowup is on export.)
The CPU runs at close to 100%. We are getting about 900mb of transfer speed up and down. Scrape count: About 72 million urls a day and after dedupe is about 30 million unique. The big key is to test the proxies before starting and then test them (clean them) when you see the speed decrease. My second cleaning is usually at about 48000 urls per minute. Only clean on one and copy the good urls to the second instance of GScraper. With each new set of proxies it is faster to stop and restart the scrape. In a couple of days I will be moving to a 16 core server with 128 GB of ram and load sharing flexibility. I'll let you know what happens.
Hangout Domination and Red-Proxies both just plain work. Avoid the other hangout product like the plague. (Guarantee not honored)