You will need more proxies that would cater to such a project more than anything else. Get a vps that supports gscraper with their proxies and you would be good to go.
i scrape with scrapebox (even using multiple instances) 24/7 using 40 private proxies on a VPS (3 3.0ghz cores and 2gb ram), and the automator plugin. I have the random delay set between 3 and 8 seconds and I dont burn up my proxies but it does take a long time scrape with so few proxies and the delays. Check out newipnow's bst. I use their proxies. whatever you buy they will double it if you post in their thread.
I own gscraper too and its lighting fast if you can afford their proxy service. I don't use it because I am on a budget (not selling services to anyone just doing my own SEO) and use those same proxies in gsa ser. Otherwise I would use gscraper and their proxy service for fast scraping of targets.
IDK if you want to scrape to find targets to post to with software or not, but heres a trick for building a massive list in conjunction with scraping, if thats what you want to scrape for
http://www.blackhatworld.com/blackh...ow-easily-build-huge-sites-lists-gsa-ser.html
The cheapest and relatively fast method is using DSL connection.
This is the workflow:
1) Set the threads to X - you need to calculate it based on your bandwidth.
2) Scrape until google blocks you, usually after a few thousand searches. You MUST NOT use cookies.
3) Disconnect/Connect your DSL modem/router to change the IP. I am doing this automatically with a simple script.
Change the user agent to IE6 - google returns content 5 times smaller compared,for example, to chrome.
You can change the user agent to some of the older browsers to return WAP(XML) content, for example -> Alcatel-BE3/1.0 UP/4.1.8h
Not sure if scrapebox can handle this with some kind of plugin, but it is a good custom solution.
I am doing this for years without any problems. Recently, i had to scrape the results from 20,000,000 searches- it took me around ~120 hours(5days) with a 12mbit DSL connection.
Really... Scrapebox?
Get Gscraper (it's a lot faster than Scrapebox) and good port scanned proxies and scrape 80-200 million urls a day. Forget all about Gscraper's proxy service as it sucks donkey balls. I'm running Gscraper along side other SEO tools on a powerful dedi, but I doubt you'll need more than a few gb ram and a core or two.
Thanks to all for answering guys!
@wowfactorinme: Their proxy subscription is good, but not as good as it used to be. I guess because more and more people starting using it. And I have other tools to scrape fast for working public proxies every day, so except the private I will use those as well, but thanks for your suggestion.
@zombie pop: Thanks a lot for your reply mate and your suggestions. How many links you scrape per day with those settings ?
Thanks for the link, I have read that thread before
I actually want to scrape targets for GSA so then I can just identify and post using GSA. Also, what VPS are you using if I may ask ?
@theMagicNumber: Those are pretty good numbers. I have the script as well to easily get new dynamic IP, however since I changed my ISP I now have a static IP which sucks, and I am not sure why my ISP does that lol since static IP-s are more expensive anyway to get. Although the user agent trick is a really good idea, thanks.
If I may ask, how many threads you use with 12 Mbit/sec connection and what are your computer's parameters ?
@Groen: I have tested them both (Gscraper the free version) and I think they are the same in terms of speed if you use the same sets of proxies. Everyone says Gscraper is faster, but that is only if you use their proxy service, which is pretty overpriced IMO.
How many proxies you use ?
