could someone explain scraping to me?

E

eqpaisley

Junior Member
Joined
Oct 16, 2012
Messages
178
Reaction score
50
So I get that there are public proxies out there and I see that some websites collect and post them. My question is, how does one come up with these lists? I have some scripting skills and would love to create a scraper but I don't even begin to know where to start.

Do you spam IP ranges and wait for a connection? If these sites that aggregate proxies didn't exist, how would one know that these public proxies were available?

EQP
 
Last edited:
Advertise on BHW
We'll you wouldn't in short. Most scrape from proxy lists like you mention. Testing billions of ips and port combos would be insane and pointless.
 
One way is to type in the IP string into google. See every website that posted this IP as a proxy and then take all of proxy lists each website has. Type in a few other IPs into google that you got from the list, then take all of those proxies, text them in scrapeox, remove dupes.
 
scan every IPv4 ip from 1.0.0.1 to 255.255.255.255 to see if they are open as proxies. There are a lot of proxies port scanners scripts out there for free.

Code: 
Ex.: [URL]http://funoverip.net/2010/11/socks-proxy-servers-scanning-with-nmap/[/URL]
 
boombap said:
We'll you wouldn't in short. Most scrape from proxy lists like you mention. Testing billions of ips and port combos would be insane and pointless.
Click to expand...

I get this, makes sense. But here do those LISTS come from? Someone, somewhere is busy pinging away, right? My goal is to find a source of fresh proxies since these public lists are so heavily trafficked.
 
Create or find A REGEX to match IPS. Create a script that downloads the sourcecode of a website and finds any Proxy IP using your REGEX and puts it in an array. Run each of these IPs through a variety of tests with whatever language you are using. In c# or vb.net I would make a web requestusing each IP. Check the timeout time and record that and catch any error to see if the proxys work or not. To test the anonymity use a php proxy judge that you can find online and host it yourself. My 2 cents.
 
As stated, one way to resolve open proxies is a tool like nmap. However you won't get very far unless you have a distributed scanning network, like what was done recently in this interesting project: (google internetcensus2012) but you still may get some hits. As to your question, scraping generally means taking an HTML page as input and using an HTML parser (e.g. JSoup, Enlive ) to extract the information you want from the page via XPATH expressions. With XPATH each element on the page has unique expression. Given a list of sites which have proxy lists, one *could* scrape through them and test programatically producing a real-time list of working proxies... hmmm.. sounds pretty useful if you need proxies.... PM me if you have any questions.
 
You can Always buy private proxies from BHW
smile.png
 
SPPChristian said:
scan every IPv4 ip from 1.0.0.1 to 255.255.255.255 to see if they are open as proxies. There are a lot of proxies port scanners scripts out there for free.

Code: 
Ex.: [URL]http://funoverip.net/2010/11/socks-proxy-servers-scanning-with-nmap/[/URL]
Click to expand...

That's 4 bilion hosts, about 64 thousand ports per host, so using that method you should get a list built by the year 2199 or something.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top