Anyone have much experience with scraping CL and the number of proxies etc needed?
I have tried various proxy sites and they end up all being\getting banned and an looking into some services that say they offer specific CL pools, but I need some idea of how many to purchase. We have also had it suggested that we buy X amazon aws IPs (I think) and use them, but the setup cost etc without knowing if it would work seems high.
Our use case is just grabbing ads, no posting, no responding, just the ads. So say ~420 cities => ~5000 ads a day.
Our old system just grabbed all the RSS feeds, and ~15->20 IPs could do that but we are seeing that the RSS feeds are spotty at best on ads.
So any input would be great. I have seen other topics here, and someone mentioned a CL forum, but I have not found it.
I have tried various proxy sites and they end up all being\getting banned and an looking into some services that say they offer specific CL pools, but I need some idea of how many to purchase. We have also had it suggested that we buy X amazon aws IPs (I think) and use them, but the setup cost etc without knowing if it would work seems high.
Our use case is just grabbing ads, no posting, no responding, just the ads. So say ~420 cities => ~5000 ads a day.
Our old system just grabbed all the RSS feeds, and ~15->20 IPs could do that but we are seeing that the RSS feeds are spotty at best on ads.
So any input would be great. I have seen other topics here, and someone mentioned a CL forum, but I have not found it.