I use Yahoo + No proxies and get knocking on a million URLs easily.
Why would you bother using proxies for harvesting URLs when Yahoo + No proxies, can be just as good, if not better then Google?
thing is though scraping with private proxies is a very bad
idea why? because try scraping 100k and watch google temp
ban your private proxies for upto 3 yrs. then your screwed
and so are your private proxies. this is why url scraping with
public - and posting with private is always the way to go
I use Yahoo + No proxies and get knocking on a million URLs easily.
Why would you bother using proxies for harvesting URLs when Yahoo + No proxies, can be just as good, if not better then Google?