1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Managing proxy pools & proxy provider recommendations

Discussion in 'Proxies' started by URetailer, May 11, 2017.

  1. URetailer

    URetailer Newbie

    Joined:
    Aug 22, 2016
    Messages:
    10
    Likes Received:
    1
    Gender:
    Male
    I have been working on a mid-scale scraping project (~=2M-3M requests/day), and have been able to get by on a small pool of fixed private proxies as well as a proxy rotation service where I route most of my requests. As time wore on, different sites have implemented different anti-scraping measures internally and with the help of various bot detection companies (akamai comes to mind), and the proxy rotation and ban detection service I was using became less and less effective, forcing me to manage an ever increasing pool of fixed dedicated IPs and essentially write my own ban detection and semi-balanced random distribution logic.

    It has gotten to the point where I should really consider significantly increasing my pool of managed IPs and ditch the service entirely. The first problem here would be budget, while private dedicated proxies are obviously preferred, I would need at least 500 ips and do not want to dish out 500/month for ips, a portion of which, I may not be able to swap out on at least a monthly basis (I do not anticipate needing to swap out many perhaps 10-50 per month, if that, while I work out some of the kinks in the ban detection and prevention logic). The other option of semi-dedicated, while being much cheaper makes things much more complicated, if not impossible, since I cannot assume the request history over any fixed period of time, the ban logic would be reduced to assuming I am the only one crawling certain domains (terrible), or possibly leaving in a small margin for error (better but far from optimal).

    My question is 2 fold:

    1) Which companies do the experienced folks here at BHW recommend that would either be able to offer bulk (500-700) proxies for under $1/per ip and allow for some swapping monthly?

    2) Or, semi-dedicated (fixed shared users, lower number would obviously be preferred) where the provider can guarantee domain restriction (I have seen providers offer dirt cheap proxies but restricted to a handful of popular domains like twitter/facebook, unfortunately I wasn't interested in those domains) so I can be assured that while sharing that connection with others, I will be the only one crawling those particular domains on that pool of proxies? (I would probably only crawl 7-15 domains depending) Failing the restriction policy, I would need an undetermined amount of swaps a few times a month, depending how many others on my shared connection are crawling the same domains.

    Thanks in advance.

    PS: I crawl all popular retailers (amazon, walmart, etc...)
    PPS: all the usual necessities - a variety of different sub-nets along with non-sequential ips, 20+ threads per proxy, headers stripped so source ip is not revealed, etc...
     
  2. LocalProxies

    LocalProxies Jr. VIP Jr. VIP

    Joined:
    Feb 15, 2016
    Messages:
    230
    Likes Received:
    36
    Occupation:
    Residential Proxy Provider
    Location:
    Worldwide Locations
    Home Page:
    Scraping is becoming increasingly difficult when you try to do it using data center proxies, your proxy ISP is the main reason why you are getting banned.

    We can offer residential proxies that work great for scraping any website. Please check our Rotating Every Request plans, they are perfect for scraping. With rotating residential proxies IP swaps are not needed, proxy swap themselves automatically and proxy network refreshes itself constantly, so even if you get bans, proxies will be replaced soon on their own. On top of that, residential proxy bans are not permanent, so even when IP is banned today, it doesn't mean it will be banned tomorrow.

    If you need more information, please contact us here or on Skype, our ID is: localproxies.
     
  3. Chris.Roark

    Chris.Roark Jr. VIP Jr. VIP

    Joined:
    Aug 16, 2016
    Messages:
    588
    Likes Received:
    147
    Home Page:
    There are certain proxy providers offering tailored solutions.

    Please check my sig. there are a few providers offering Shared proxies for scraping.

    If you need help just PM me
     
  4. URetailer

    URetailer Newbie

    Joined:
    Aug 22, 2016
    Messages:
    10
    Likes Received:
    1
    Gender:
    Male
    Your service sounds great, unfortunately the lowest cost plan I am seeing for US rotating proxies (every request) is 2k/month, which as I stated in my original post is well out of my budget. Did I miss any of your plans?
     
  5. akssiv2007

    akssiv2007 Senior Member

    Joined:
    Jul 11, 2013
    Messages:
    897
    Likes Received:
    198
    Gender:
    Male
    Occupation:
    Webber
    Location:
    Earth
    Op you should check p2p proxies service in marketplace.

    https://www.blackhatworld.com/seo/p...mium-proxies-200k-pool-single-gateway.821791/

    Their pool is large enough for scraping data. Though they have a different pricing model $2/GB worldwide pool
    $4/GB usa pool

    So they charge for bandwidth consumed and it's not monthly. There is no expiration.

    I have found them better over few others that I have used previously, it's pay as you go.
    It can be costly if your target page has lots of images.

    I mostly use it for G scraping and works fine, the USA pool is great to create social accounts also.

    You can give this a try maybe this helps you.