uchiha.jain
Regular Member
- Sep 14, 2009
- 295
- 84
Hi friends, I need your help with selecting proxies for an Amazon price scraper for my repricer.
A bit of background on the stack:
The app is built on NodeJS and I'm using ZombieJS for scraping (it's a barebones headless browser that simply allows the websites' JS to be run and cookies to be set).
About proxies, I really don't see a point in using dedicated proxies as IPs will eventually get banned and will need to replaced. It's an expensive solution all in all. So rotating proxies it is for us. I've tried out a few proxy services. Here's my experience with them:
- Gimmeproxy (not exactly rotating/backconnect proxies but close): I used their free plan to test them out but found their proxies working only about half the time. Trouble is, when the proxy doesn't work, it will time out and the lag wouldn't be acceptable with the repricer.
- Stormproxies: I tested with their basic plan and found most of their IPs to be already banned by Amazon (503 error). Moreover, it seems the owners lie about having 70k proxies in rotation when actually they have far less (they also seem to want to handle refunds regarding this in private which seems shady as fuck) and I have my doubts about even those being residential IPs given that most if not all of their IPs were already banned by Amazon. So we won't do any business with them unless they can prove these claims wrong.
- Proxymesh: This service worked out the best for us. They limit the accounts by bandwidth instead of number of parallel connections (like stormproxies) which worked well for our use case. Currently we use their $20 plan which only gives us access to 20 proxies at a time and now that we're starting to scale they seem to start throwing errors very frequently (The success rate varies from 1%-30% at times). We make it work by retrying the request which unlike gimmeproxy works here since it's returning 503 from Amazon instead of timing out but we really need an alternative badly. Their $50 plan seems to offer access to 25k proxies (Haven't tested if it's true, does anyone know for sure?) which might well work out for us at our current scale.
So this is where we're at right now and normally I would simply go with proxymesh $50 plan since I think it will work for us at a reasonable price. But I keep reading about IPv6 proxies and how they're being sold for pennies and everybody seems to be getting good results from them. Here are a few services I found but I don't really find them to be much cheaper than proxymesh (at least at our current scale).
for ex.
https://oxylabs.io/pricing
https://www.infiniproxy.com/pricing.php (best combination of price + number of IPs, I think. What does unlimited IPs mean here though, truly?)
http://ipv6proxies.com/
https://proxy6.net/en/price
https://v6proxies.com/#section-pricing
https://proxy-seller.com/ipv6.html#rates
Also, while the IPv6 proxy vendors bill their products as offering thousands and millions of IPs, it seems most websites follow the policy of banning a /64 block in it's entirety. Does that mean they are no better than IPv4 proxies? Also, just curious, how many IPs are there in a /64 block? A /24 block of IPv4 contains 256 IPs, am I right?
Sorry about so many questions but I have limited understanding of the topic and the more dig, the more questions I find. Any tips about scraping from Amazon are also much appreciated. Thanks for taking the time to read and looking forward to your advice.
Cheers,
Jain
A bit of background on the stack:
The app is built on NodeJS and I'm using ZombieJS for scraping (it's a barebones headless browser that simply allows the websites' JS to be run and cookies to be set).
About proxies, I really don't see a point in using dedicated proxies as IPs will eventually get banned and will need to replaced. It's an expensive solution all in all. So rotating proxies it is for us. I've tried out a few proxy services. Here's my experience with them:
- Gimmeproxy (not exactly rotating/backconnect proxies but close): I used their free plan to test them out but found their proxies working only about half the time. Trouble is, when the proxy doesn't work, it will time out and the lag wouldn't be acceptable with the repricer.
- Stormproxies: I tested with their basic plan and found most of their IPs to be already banned by Amazon (503 error). Moreover, it seems the owners lie about having 70k proxies in rotation when actually they have far less (they also seem to want to handle refunds regarding this in private which seems shady as fuck) and I have my doubts about even those being residential IPs given that most if not all of their IPs were already banned by Amazon. So we won't do any business with them unless they can prove these claims wrong.
- Proxymesh: This service worked out the best for us. They limit the accounts by bandwidth instead of number of parallel connections (like stormproxies) which worked well for our use case. Currently we use their $20 plan which only gives us access to 20 proxies at a time and now that we're starting to scale they seem to start throwing errors very frequently (The success rate varies from 1%-30% at times). We make it work by retrying the request which unlike gimmeproxy works here since it's returning 503 from Amazon instead of timing out but we really need an alternative badly. Their $50 plan seems to offer access to 25k proxies (Haven't tested if it's true, does anyone know for sure?) which might well work out for us at our current scale.
So this is where we're at right now and normally I would simply go with proxymesh $50 plan since I think it will work for us at a reasonable price. But I keep reading about IPv6 proxies and how they're being sold for pennies and everybody seems to be getting good results from them. Here are a few services I found but I don't really find them to be much cheaper than proxymesh (at least at our current scale).
for ex.
https://oxylabs.io/pricing
https://www.infiniproxy.com/pricing.php (best combination of price + number of IPs, I think. What does unlimited IPs mean here though, truly?)
http://ipv6proxies.com/
https://proxy6.net/en/price
https://v6proxies.com/#section-pricing
https://proxy-seller.com/ipv6.html#rates
Also, while the IPv6 proxy vendors bill their products as offering thousands and millions of IPs, it seems most websites follow the policy of banning a /64 block in it's entirety. Does that mean they are no better than IPv4 proxies? Also, just curious, how many IPs are there in a /64 block? A /24 block of IPv4 contains 256 IPs, am I right?
Sorry about so many questions but I have limited understanding of the topic and the more dig, the more questions I find. Any tips about scraping from Amazon are also much appreciated. Thanks for taking the time to read and looking forward to your advice.
Cheers,
Jain
