1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Does anybody else have this problem when scraping/checking proxies?

Discussion in 'Proxies' started by phr0zt, Mar 29, 2012.

  1. phr0zt

    phr0zt Regular Member

    Joined:
    Jun 30, 2007
    Messages:
    439
    Likes Received:
    182
    So I have a few VPS's to run my tools on, but I also have a badass gaming desktop that I like to let run constantly.

    But I have this problem (on the desktop computer):

    When I run proxy scraping and checking tools, such as ZennoPoster's Proxy Checker, ProxyGoblin, and the built-in proxy tool for Search Engine Ranker, after a while my internet locks up and I can't do anything with it.

    I can't navigate to websites, use my tools, or anything. (Google and other sites fail to load, Search Engine Ranker starts saying "unable to download" for every site it finds, etc.)

    I only use one tool at a time, and only run about 40 threads or so... I'm on a 15mbps internet connection, so I feel that it shouldn't be using up ALL of it.

    I've used TCPOptimizer and tried multiple settings, including "Optimal," and it doesn't seem to help.

    Does anyone else have this problem? Anyone know a solution?

    Thanks!
     
  2. proxygo

    proxygo Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 2, 2008
    Messages:
    10,244
    Likes Received:
    8,701
    im guessing your maxing out your connection
    if pages dont load - thats usually a sign.

    i have 10 meg connection and if i set my scrapers
    on to many cons pages wont load.
    but the thing is, i dont scrape for proxies anymore

    i port scan using 3000 connections and never have
    an issue collecting public proxies that way + they last
    longer less used.
     
    • Thanks Thanks x 1
    Last edited: Mar 29, 2012
  3. phr0zt

    phr0zt Regular Member

    Joined:
    Jun 30, 2007
    Messages:
    439
    Likes Received:
    182
    cool thanks for the reply!

    any suggestions for good port scanning tools to find proxies?
     
  4. proxygo

    proxygo Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 2, 2008
    Messages:
    10,244
    Likes Received:
    8,701
    i use proxyfire to pull proxies from my ip range lists
    i use a single ip port scanner to draw the proxies out
    which proxyfire converts to ip lists

    the trick is, knowing where to scan and on what ports
    u wont learn that in 5 minutes or 5 months, it takes
    days and weeks just to find a few good sources .
     
    • Thanks Thanks x 1
  5. Sophomoric

    Sophomoric Junior Member

    Joined:
    Aug 4, 2011
    Messages:
    116
    Likes Received:
    9
    Diversify your proxy source (proxy forums etc) and tweak your harvest settings (connections and timeout) until you find a sweet spot.
     
    • Thanks Thanks x 1
  6. DrEvil123

    DrEvil123 Junior Member

    Joined:
    Dec 31, 2011
    Messages:
    166
    Likes Received:
    141
    Cut your connections in half.

    I normally run about 15-20 on a similar line, when I go higher then I start seeing some false failures.
     
    • Thanks Thanks x 1
  7. proxygo

    proxygo Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 2, 2008
    Messages:
    10,244
    Likes Received:
    8,701
    prob just drop cons a bit
     
    • Thanks Thanks x 1
  8. phr0zt

    phr0zt Regular Member

    Joined:
    Jun 30, 2007
    Messages:
    439
    Likes Received:
    182
    interesting, thanks for your input everyone. I'll start with 20 connections and see how that works, it just seems weird that I can't run a few more simultaneously on this connection speed. Oh well, when I move to my new apt soon I'll be investing in wideband, that should help out some haha. Also gonna try to figure out how to pull off what proxygo said.

    Thanks again everyone!
     
    Last edited: Mar 30, 2012