1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How do you scrape and filter proxies that will last?

Discussion in 'BlackHat Lounge' started by karma05, Sep 18, 2016.

  1. karma05

    karma05 Regular Member

    Joined:
    May 30, 2013
    Messages:
    384
    Likes Received:
    21
    Location:
    Ramsdale
    I need some public proxies for a software, but it won't work if they keep expiring quickly. So is there a way in Scrapebox or something to filter down to the ones which will stay alive for quite a long time?
     
  2. HoNeYBiRD

    HoNeYBiRD Jr. VIP Jr. VIP

    Joined:
    May 1, 2009
    Messages:
    7,506
    Likes Received:
    8,434
    Gender:
    Male
    Occupation:
    Geographer, Tourism Manager
    Location:
    Ghosted
    You found the problem with public proxies: they are unreliable and don't last.
    When you finished scraping and checking, some of your proxies won't work already even if they worked, when you checked them. Some proxies will never work again, others are only down temporarily and will work later.
    No-one is able to tell you how long will a public proxy last, hence there's no tool with filtering options for this.
     
  3. karma05

    karma05 Regular Member

    Joined:
    May 30, 2013
    Messages:
    384
    Likes Received:
    21
    Location:
    Ramsdale
    What if you tested a scraped list all day long, wouldn't the ones which lasted all day be likely to last for quite a while?
     
  4. HoNeYBiRD

    HoNeYBiRD Jr. VIP Jr. VIP

    Joined:
    May 1, 2009
    Messages:
    7,506
    Likes Received:
    8,434
    Gender:
    Male
    Occupation:
    Geographer, Tourism Manager
    Location:
    Ghosted
    That's a fair point, although you still couldn't possibly know their lifetime, it can be a couple more hours, a few days or just a minute etc. It's only an assumption, that if they work for a day, they will continue to do so for a while. When you test a proxy, you don't monitor the uptime, you only know that the proxy is alive or dead in the very moment when you tested it. If it's dead, it could be up in the next second or dead for good, if it's up, it could have been dead in the second before that or it could be dead in the next second.

    To determine at least a somewhat approximate uptime, you would need to test the same proxies a couple of times per day. Depending on the amount you need, it can take quite some time and might not worth the hassle at all. They're still public proxies, even when they work, most of them is sluggish as hell.

    There are cheaper type of proxies than private dedicated or shared proxies. Those are called backconnect proxies, where you pay for the amount of ports, not individual proxies. You have a big enough pool of proxies and the proxies rotate in a given timeframe. If you have a little capital, it might worth to invest in a service like that. So you can spare yourself the time what scraping and testing takes from you.
     
    • Thanks Thanks x 1
  5. akacash

    akacash Jr. VIP Jr. VIP

    Joined:
    Jan 16, 2010
    Messages:
    844
    Likes Received:
    622
    Location:
    The Beach, USA
    How long do you need the proxies to stay online for? What Honeybird said was correct in that there's no way to know how long a public proxy will last, even if you tracked it for a week and it was online the entire time, that's not to say it wouldnt go off 2 seconds later. If you need to use the proxies for more than a few minutes you're going to always have that problem with public proxies. You could try port scanning some and you might get better results, but buying a small package would probably be best for you if you need them to be somewhat stable. Good luck though.
     
    • Thanks Thanks x 1
  6. The Curator

    The Curator Senior Member

    Joined:
    Dec 27, 2013
    Messages:
    1,054
    Likes Received:
    443
    Here's a tutorial on how to have your SB continually scrape and harvest proxies
     
  7. conrulez

    conrulez Power Member

    Joined:
    Dec 29, 2009
    Messages:
    539
    Likes Received:
    429
    Gender:
    Male
    Location:
    USA
    Your MOST valuable asset is TIME. Stop wasting it with public BS proxies. Get up and go buy some from buyproxies.org or someone highly reputable like them. If you prefer to waste time and money with public crap, go ahead. If you want serious results, you need to act like a serious professional. NOBODY who is seeking or producing professional results EVER uses public proxies in the manner you are talking about.

    Life's about choices. Choose wisely. Good luck.
     
  8. JustUs

    JustUs Power Member

    Joined:
    May 6, 2012
    Messages:
    626
    Likes Received:
    588
    Go buy private proxies - amusing. Public proxies will always fail - amusing.

    Too much generalization!
    Buy a package of proxies and then run them through a spam checker. Many will have failed because they have been used in the past.
    Scrape up a few hundred proxies proxies and run them through a spam checker. Many will fail and some will pass.

    The case is the same in both instances. Some will fail and some will pass. There is simply no way to get proxies that will always be up and will always pass a spam test.

    I both scrape and port test. While I have some port tested proxies that have been up for months and pass spam tests, I also have publicly tested proxies that have been up for months and pass spam tests. The difference is that the port tested proxies generally are alive more consistently because they are 1. less commonly used, 2. more often are government IP's in different countries, and 3. many are associated with trackers.

    Now read the large number of threads about private proxies not working with some tool - more often than not Scrapebox. The advise is to get a different package of proxies because the private proxies are banned by a search engine.

    Indeed, the chicken and the egg.