I have been working on a mid-scale scraping project (~=2M-3M requests/day), and have been able to get by on a small pool of fixed private proxies as well as a proxy rotation service where I route most of my requests. As time wore on, different sites have implemented different anti-scraping measures internally and with the help of various bot detection companies (akamai comes to mind), and the proxy rotation and ban detection service I was using became less and less effective, forcing me to manage an ever increasing pool of fixed dedicated IPs and essentially write my own ban detection and semi-balanced random distribution logic.
It has gotten to the point where I should really consider significantly increasing my pool of managed IPs and ditch the service entirely. The first problem here would be budget, while private dedicated proxies are obviously preferred, I would need at least 500 ips and do not want to dish out 500/month for ips, a portion of which, I may not be able to swap out on at least a monthly basis (I do not anticipate needing to swap out many perhaps 10-50 per month, if that, while I work out some of the kinks in the ban detection and prevention logic). The other option of semi-dedicated, while being much cheaper makes things much more complicated, if not impossible, since I cannot assume the request history over any fixed period of time, the ban logic would be reduced to assuming I am the only one crawling certain domains (terrible), or possibly leaving in a small margin for error (better but far from optimal).
My question is 2 fold:
1) Which companies do the experienced folks here at BHW recommend that would either be able to offer bulk (500-700) proxies for under $1/per ip and allow for some swapping monthly?
2) Or, semi-dedicated (fixed shared users, lower number would obviously be preferred) where the provider can guarantee domain restriction (I have seen providers offer dirt cheap proxies but restricted to a handful of popular domains like twitter/facebook, unfortunately I wasn't interested in those domains) so I can be assured that while sharing that connection with others, I will be the only one crawling those particular domains on that pool of proxies? (I would probably only crawl 7-15 domains depending) Failing the restriction policy, I would need an undetermined amount of swaps a few times a month, depending how many others on my shared connection are crawling the same domains.
Thanks in advance.
PS: I crawl all popular retailers (amazon, walmart, etc...)
PPS: all the usual necessities - a variety of different sub-nets along with non-sequential ips, 20+ threads per proxy, headers stripped so source ip is not revealed, etc...
It has gotten to the point where I should really consider significantly increasing my pool of managed IPs and ditch the service entirely. The first problem here would be budget, while private dedicated proxies are obviously preferred, I would need at least 500 ips and do not want to dish out 500/month for ips, a portion of which, I may not be able to swap out on at least a monthly basis (I do not anticipate needing to swap out many perhaps 10-50 per month, if that, while I work out some of the kinks in the ban detection and prevention logic). The other option of semi-dedicated, while being much cheaper makes things much more complicated, if not impossible, since I cannot assume the request history over any fixed period of time, the ban logic would be reduced to assuming I am the only one crawling certain domains (terrible), or possibly leaving in a small margin for error (better but far from optimal).
My question is 2 fold:
1) Which companies do the experienced folks here at BHW recommend that would either be able to offer bulk (500-700) proxies for under $1/per ip and allow for some swapping monthly?
2) Or, semi-dedicated (fixed shared users, lower number would obviously be preferred) where the provider can guarantee domain restriction (I have seen providers offer dirt cheap proxies but restricted to a handful of popular domains like twitter/facebook, unfortunately I wasn't interested in those domains) so I can be assured that while sharing that connection with others, I will be the only one crawling those particular domains on that pool of proxies? (I would probably only crawl 7-15 domains depending) Failing the restriction policy, I would need an undetermined amount of swaps a few times a month, depending how many others on my shared connection are crawling the same domains.
Thanks in advance.
PS: I crawl all popular retailers (amazon, walmart, etc...)
PPS: all the usual necessities - a variety of different sub-nets along with non-sequential ips, 20+ threads per proxy, headers stripped so source ip is not revealed, etc...
