We need to spread 10 TB of data per month across 750M requests to crawl some websites. The IPs must be from a wide variety of subnets from residential areas.
We would be happy doubling down on that to crawl twice as fast if you’re able to deliver twice the throughput or more, provided of course...
Well, you just have to experiment. You might put in a ton of effort into something without it working out, and then something else does. It`s just the way of IM.
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.