1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How many connections for public proxies in Scrapebox?

Discussion in 'Black Hat SEO Tools' started by rodyaj, Sep 8, 2012.

  1. rodyaj

    rodyaj Newbie

    Joined:
    May 16, 2012
    Messages:
    23
    Likes Received:
    5
    On a 100mbit VPS what is the max connections I should set for URL harvesting on scraped public proxies? They seem to burn out within about 10 minutes if I go much above 100 in the max connections. Should I just try to add as many proxies as I have connections, or will I really need to lower the connections to stop them burning out so fast? What about timeout settings?

    Some people recommend to buy private proxies and set lower max connections (e.g., 5) and timeout values to stop burn out. Will private proxies still scrape as fast on lower max connections?

    Eventually I want to start using a tool like yaph to portscan for my own proxies so I can just keep adding them in bulk without caring about burn-out, but it looks like it will take a lot of trial and error to go through all the known country cidrs and find some good ranges. That said, I just want to make do with public proxies for the time being.
     
  2. Scritty

    Scritty Elite Member Premium Member

    Joined:
    May 1, 2010
    Messages:
    2,807
    Likes Received:
    4,496
    Occupation:
    Affiliate Marketer
    Location:
    UK
    Home Page:
    They are public. So there is you and maybe 200 other people using them.

    Whatever setting you make will on average make 1/2 a percent difference to the overall workload those proxies are doing. (1 in 200 = 1/2 a percent)
    I would use them as quickly as you can. If you don't burn them out - the other 199 people using them certainly will.
    Might as well get as much juice out of them as you can before that happens.
     
    • Thanks Thanks x 1
  3. HelloInsomnia

    HelloInsomnia Jr. Executive VIP Jr. VIP Premium Member

    Joined:
    Mar 1, 2009
    Messages:
    1,817
    Likes Received:
    2,914
    Private proxies will scrape faster even on less connections. Especially if you count the time saved when public ones burnout then you have to scrape and test more. Private is the way to go even if you only have 5 connections running (you will need 30+ to do this).
     
    • Thanks Thanks x 2
  4. rubyakerman

    rubyakerman Newbie

    Joined:
    Jun 21, 2012
    Messages:
    20
    Likes Received:
    0
    i got a bann from that
     
  5. HelloInsomnia

    HelloInsomnia Jr. Executive VIP Jr. VIP Premium Member

    Joined:
    Mar 1, 2009
    Messages:
    1,817
    Likes Received:
    2,914
    On your proxies? Then you were probably using too many connections. If your running private you don't need many to scrape fast. Of course having more proxies and less connections will keep you from getting banned.
     
  6. Scritty

    Scritty Elite Member Premium Member

    Joined:
    May 1, 2010
    Messages:
    2,807
    Likes Received:
    4,496
    Occupation:
    Affiliate Marketer
    Location:
    UK
    Home Page:
    These days I use 10 proxies minimum for ONE THREAD.
    With Yahoo for some reason it's 20 per thread to avoid 999 errors.

    3 years ago those ratios were the other way around.

    Scritty
     
    • Thanks Thanks x 2