1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Proxy Check bottleneck problem (over 300,000 daily)

Discussion in 'Proxies' started by timothywcrane, Jun 6, 2011.

  1. timothywcrane

    timothywcrane Power Member

    Joined:
    Apr 25, 2009
    Messages:
    590
    Likes Received:
    236
    Occupation:
    Internet Promotion Management
    Location:
    USA
    Home Page:
    I have finally got to the point that scraping proxies is as simple as pie for me, but I have a serious problem. I get about 3-400 thousand proxies on a daily basis that I then have to check, and there is no way in Nebraska that I can ever get them sorted and checked to use them before they go bad. I tried using Charon and accessdiver, but it is way too slow, even on a cable connection. Is there any way to speed this up? Have thought of using online proxy checkers, but they will just share them all with others killing them faster. Any ideas?
    121
     
  2. nettlestein

    nettlestein Registered Member

    Joined:
    Feb 27, 2010
    Messages:
    66
    Likes Received:
    13
    Occupation:
    owner of linklabia.info and student at unlv
    Location:
    las vegas
    Home Page:
    man here in vegas we have cox with 50mbit DL speed and i crank out good but i use modified sicksubmitter proxy tool it move fast on my line. try it out
     
  3. mazgalici

    mazgalici Supreme Member

    Joined:
    Jan 2, 2009
    Messages:
    1,489
    Likes Received:
    881
    Home Page:
    you sould split that on multiple servers
     
  4. proxygo

    proxygo Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 2, 2008
    Messages:
    10,296
    Likes Received:
    8,717
    might wanna try only scrape using scrapebox setting
    known as time and set to last 24hrs - that will bring it
    down to the most recent. at most u will get 5-6k proxies
    to test. if u have it to 3-400k u havnt got it down yet
     
  5. Swiss

    Swiss Power Member

    Joined:
    Jun 3, 2011
    Messages:
    551
    Likes Received:
    323
    Location:
    Take a guess
    What's usually left over after testing those 400K proxies, removing dups.? Narrow down your sources to the ones that offer low latency proxies and aren't easily findable. Max out your connections, test some and adjust the settings accordingly.
    I always have 100 connections at once..
     
  6. timothywcrane

    timothywcrane Power Member

    Joined:
    Apr 25, 2009
    Messages:
    590
    Likes Received:
    236
    Occupation:
    Internet Promotion Management
    Location:
    USA
    Home Page:
    Thanks for all of the advice. Unfortunately I do not have scrapebox, I am just using a google footprint scraper and then using that to leech in AccessDiver, then using Charon to test (works better than AD). The 300k is after removing dupes from the scrape. I am definately taking the advice to spread the checking out over more platforms, but I am still limited by bandwidth, as all of my machines only have one Internet connection. I may just have to suffer through it for now and get a VPS and scrapebox ASAP. Thanks again.
    420
     
  7. proxygo

    proxygo Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 2, 2008
    Messages:
    10,296
    Likes Received:
    8,717
    bring your foot print time down to
    the last 24hrs - if you set no time u will
    end scraping proxies from yrs ago trust me
    ive been in the public proxie selling game
    7 yrs - lower your time stamp