Proxy Check bottleneck problem (over 300,000 daily)

timothywcrane · Jun 6, 2011

I have finally got to the point that scraping proxies is as simple as pie for me, but I have a serious problem. I get about 3-400 thousand proxies on a daily basis that I then have to check, and there is no way in Nebraska that I can ever get them sorted and checked to use them before they go bad. I tried using Charon and accessdiver, but it is way too slow, even on a cable connection. Is there any way to speed this up? Have thought of using online proxy checkers, but they will just share them all with others killing them faster. Any ideas?
121

nettlestein · Jun 6, 2011

man here in vegas we have cox with 50mbit DL speed and i crank out good but i use modified sicksubmitter proxy tool it move fast on my line. try it out

mazgalici · Jun 6, 2011

you sould split that on multiple servers

proxygo · Jun 6, 2011

might wanna try only scrape using scrapebox setting
known as time and set to last 24hrs - that will bring it
down to the most recent. at most u will get 5-6k proxies
to test. if u have it to 3-400k u havnt got it down yet

Swiss · Jun 6, 2011

timothywcrane said:
I have finally got to the point that scraping proxies is as simple as pie for me, but I have a serious problem. I get about 3-400 thousand proxies on a daily basis that I then have to check, and there is no way in Nebraska that I can ever get them sorted and checked to use them before they go bad. I tried using Charon and accessdiver, but it is way too slow, even on a cable connection. Is there any way to speed this up? Have thought of using online proxy checkers, but they will just share them all with others killing them faster. Any ideas?
121

What's usually left over after testing those 400K proxies, removing dups.? Narrow down your sources to the ones that offer low latency proxies and aren't easily findable. Max out your connections, test some and adjust the settings accordingly.
I always have 100 connections at once..

timothywcrane · Jun 6, 2011

Thanks for all of the advice. Unfortunately I do not have scrapebox, I am just using a google footprint scraper and then using that to leech in AccessDiver, then using Charon to test (works better than AD). The 300k is after removing dupes from the scrape. I am definately taking the advice to spread the checking out over more platforms, but I am still limited by bandwidth, as all of my machines only have one Internet connection. I may just have to suffer through it for now and get a VPS and scrapebox ASAP. Thanks again.
420

proxygo · Jun 6, 2011

bring your foot print time down to
the last 24hrs - if you set no time u will
end scraping proxies from yrs ago trust me
ive been in the public proxie selling game
7 yrs - lower your time stamp

Proxy Check bottleneck problem (over 300,000 daily)

timothywcrane

Supreme Member

nettlestein

Registered Member

mazgalici

Supreme Member

proxygo

Elite Member

Swiss

Power Member

timothywcrane

Supreme Member

proxygo

Elite Member

Main Menu

Marketplace

Making Money

BlackHat World