I've always scraped Yahoo only because of the need for proxies when scraping Google, but for some reason Yahoo is messing up with me giving 403 errors all over the place. I am wondering if this is due to the Yahoo API change which is due, er, today.
Anyway, whatever the case, I need to scrape Google as Yahoo doesn't appear to allow special characters such as those with accents. Or as I said it may just be Yahoo shutting down altogether.
For Google I need proxies but I have always found that I get a REALLY shitty success rate in terms of the number of keywords successfully completed at the end of a harvest, even with the excellent proxies from proxygo, then tested/filtered in SB and high latency proxies removed. The public proxies just seem to die to easily. Even when I use around 600 working proxies all with latency of under 1000 (!! ) I still get most of my keywords in the red (not completed).
Now my question is, if a proxy dies mid-keyword, or even at the beginning of a keyword, shouldn't the proxy be rotated and the same keyword be scraped again, until completion? Because I seem to be getting a hell of a lot of red 'not completed' keywords in Google when this darn thing should be rotating. How can you say the harvest is 'completed' when 85% of the keywords are not done due to proxies either dying or timing out?
Anyway, whatever the case, I need to scrape Google as Yahoo doesn't appear to allow special characters such as those with accents. Or as I said it may just be Yahoo shutting down altogether.
For Google I need proxies but I have always found that I get a REALLY shitty success rate in terms of the number of keywords successfully completed at the end of a harvest, even with the excellent proxies from proxygo, then tested/filtered in SB and high latency proxies removed. The public proxies just seem to die to easily. Even when I use around 600 working proxies all with latency of under 1000 (!! ) I still get most of my keywords in the red (not completed).
Now my question is, if a proxy dies mid-keyword, or even at the beginning of a keyword, shouldn't the proxy be rotated and the same keyword be scraped again, until completion? Because I seem to be getting a hell of a lot of red 'not completed' keywords in Google when this darn thing should be rotating. How can you say the harvest is 'completed' when 85% of the keywords are not done due to proxies either dying or timing out?