Massive proxy scraping and validation (200k proxies)

[r1ch11r1ch]

I am in the process of writing a proxy scraper that scrapes proxies from various sources (GitHub, webpages, APIs, google Dorking) and validates them.
Currently, I am getting 200k unique proxies.

The goal is to validate and rank them based on latency, bandwidth, location, privacy and other metrics, so a user can get the fastest nearby proxies for free.
It should be something like ProxyBroker2, but better
I am also planning on adding a webapp, as the tool runs only from the CLI right now.

Happy to hear some thoughts and insights!

 

[pythonicable]

I don’t know, but scraped proxies usually seem to be shit for me.. Botnetted ones are usually better. Of course datacenter ones are shit, but if for privacy solely then I guess scraped ones could work.

 

[r1ch11r1ch]

I don’t know, but scraped proxies usually seem to be shit for me.. Botnetted ones are usually better. Of course datacenter ones are shit, but if for privacy solely then I guess scraped ones could work.

Yes, scraped proxies will have not the best quality. The use case will mostly be for scraping and privacy.

 

[saferidge]

That sounds like a great project! One challenge is that public proxies can die off pretty quickly, so keeping the list fresh might be a bit tricky. It’d be smart to set up a system that constantly checks and weeds out the dead ones.

 

[r1ch11r1ch]

That sounds like a great project! One challenge is that public proxies can die off pretty quickly, so keeping the list fresh might be a bit tricky. It’d be smart to set up a system that constantly checks and weeds out the dead ones.

Yes, I have implemented it with a SQLite database and cronjobs, so that proxies are rechecked regularly

 

[Panther28]

Happy to hear some thoughts and insights!

I used to get about 100-200k max when I was using a proxy software that I fed proxygo's proxy lists into to, as many of them would come back up after first dead, so I would just loop the list and check every 5 minutes.

The problem is that worldwide there are only a few thousand proxies every available at one time. Once they die out that's it, your connection is lost, and most of the time your lucky that they are lasting for more than 3-5 minutes, most time its less than a minute.

Someone on BHW used to sell PHP Linux software that would scan every IP address on the net in about 15 minutes, lol, but you had to get a server in a country that wouldn't bother with all the reports of "stop pinging our address". lol

They had a business of listing their proxies for members on their site. But they only lasted a couple of years then went bust or had complete legal problems. As the site and service and product is no more now, and there was never a product as good as that.

 

[CreditNimbus]

I am in the process of writing a proxy scraper that scrapes proxies from various sources (GitHub, webpages, APIs, google Dorking) and validates them.
Currently, I am getting 200k unique proxies.

The goal is to validate and rank them based on latency, bandwidth, location, privacy and other metrics, so a user can get the fastest nearby proxies for free.
It should be something like ProxyBroker2, but better
I am also planning on adding a webapp, as the tool runs only from the CLI right now.

Happy to hear some thoughts and insights!

Your project sounds promising! A few suggestions to improve it:

1. Validation & Filtering – Implement real-time checking to remove dead or slow proxies quickly.
2. Scoring System – Rank proxies based on uptime, speed, and anonymity level (elite, anonymous, transparent).
3. Geo-Filtering – Allow users to choose proxies based on location for better performance.
4. Rotating Proxies – Add an auto-rotation feature to prevent bans and rate limits.
5. WebApp UI – A simple dashboard with filters (country, speed, type) will improve usability.
6. API Integration – Offer an API for users who want to automate proxy retrieval.

If done right, this could be a solid alternative to ProxyBroker2 with better real-time updates and usability. Keep us posted!

To validate a proxy, follow these steps:

1. Check Connectivity – Try sending a request through the proxy to a known website (e.g., https://www.google.com). If it responds, the proxy is working.
2. Measure Latency – Record the response time to determine how fast the proxy is.
3. Check Anonymity Level – Use services like https://ip-api.com or https://httpbin.org/ip to see if the proxy leaks your real IP.
4. Verify Location – Compare the proxy’s IP with geolocation databases (e.g., MaxMind or IPInfo).
5. Test Bandwidth – Download a small file or use speed tests to measure proxy performance.
6. Check for Blacklisting – Use online databases (Spamhaus, AbuseIPDB) to see if the proxy is flagged as malicious.
7. Protocol Support – Test if the proxy works with HTTP, HTTPS, SOCKS4, or SOCKS5 based on your needs.

Automate these checks in your scraper to filter out bad proxies efficiently!

 

[PsoyNe]

Can you tell me, are we talking about mobile proxies or resident ones?

 

[Proxy Cake]

Your project sounds promising! A few suggestions to improve it:

1. Validation & Filtering – Implement real-time checking to remove dead or slow proxies quickly.
2. Scoring System – Rank proxies based on uptime, speed, and anonymity level (elite, anonymous, transparent).
3. Geo-Filtering – Allow users to choose proxies based on location for better performance.
4. Rotating Proxies – Add an auto-rotation feature to prevent bans and rate limits.
5. WebApp UI – A simple dashboard with filters (country, speed, type) will improve usability.
6. API Integration – Offer an API for users who want to automate proxy retrieval.

If done right, this could be a solid alternative to ProxyBroker2 with better real-time updates and usability. Keep us posted!

To validate a proxy, follow these steps:

1. Check Connectivity – Try sending a request through the proxy to a known website (e.g., https://www.google.com). If it responds, the proxy is working.
2. Measure Latency – Record the response time to determine how fast the proxy is.
3. Check Anonymity Level – Use services like https://ip-api.com or https://httpbin.org/ip to see if the proxy leaks your real IP.
4. Verify Location – Compare the proxy’s IP with geolocation databases (e.g., MaxMind or IPInfo).
5. Test Bandwidth – Download a small file or use speed tests to measure proxy performance.
6. Check for Blacklisting – Use online databases (Spamhaus, AbuseIPDB) to see if the proxy is flagged as malicious.
7. Protocol Support – Test if the proxy works with HTTP, HTTPS, SOCKS4, or SOCKS5 based on your needs.

Automate these checks in your scraper to filter out bad proxies efficiently!

Dude chill out on Chatgpt

 

[ProxyPapa]

I am in the process of writing a proxy scraper that scrapes proxies from various sources (GitHub, webpages, APIs, google Dorking) and validates them.

You could start by posting list of scraped proxies on Github every hour or so. As for use case for these proxies, I am not sure I would ever touch them. Most of them will be either dead or slow and will not last. Security is another major concern

 

[ristoriel]

I am in the process of writing a proxy scraper that scrapes proxies from various sources (GitHub, webpages, APIs, google Dorking) and validates them.
Currently, I am getting 200k unique proxies.

The goal is to validate and rank them based on latency, bandwidth, location, privacy and other metrics, so a user can get the fastest nearby proxies for free.
It should be something like ProxyBroker2, but better
I am also planning on adding a webapp, as the tool runs only from the CLI right now.

Happy to hear some thoughts and insights!

Ranking proxies based on real-time performance metrics could make it a game-changer. A web app would definitely make it more accessible—are you planning to add any filtering or automation features for users?

 

[ProxyPapa]

Ranking proxies based on real-time performance metrics could make it a game-changer.

Your response sounds like AI generated blurb. Real time performance? Which metrics will you be using to measure it? No, just ping is not an adequate measure. And how are you even implementing this on a proxy used by possibly hundreds of users? This would be resource intensive task and not worth it for public proxy that will disappear in next hour or so.

 

[r1ch11r1ch]

Thanks for the feedback!

I used to get about 100-200k max when I was using a proxy software that I fed proxygo's proxy lists into to, as many of them would come back up after first dead, so I would just loop the list and check every 5 minutes.

The problem is that worldwide there are only a few thousand proxies every available at one time. Once they die out that's it, your connection is lost, and most of the time your lucky that they are lasting for more than 3-5 minutes, most time its less than a minute.

Someone on BHW used to sell PHP Linux software that would scan every IP address on the net in about 15 minutes, lol, but you had to get a server in a country that wouldn't bother with all the reports of "stop pinging our address". lol

They had a business of listing their proxies for members on their site. But they only lasted a couple of years then went bust or had complete legal problems. As the site and service and product is no more now, and there was never a product as good as that.

Yes, I will have to feed the proxies into a proxy server, which allows me then to regularly rotate them and expose a single proxy which will make further integration in other programs easier. I already thought about internet scanning with sth like masscan, could be added sometimes in the future (not a priority rn though).

Do you remember how much money he made/ does the listing or posts of him still exist?

Can you tell me, are we talking about mobile proxies or resident ones?

Well as they are scraped you will get all types (not the best though as they are readily available on the internet). mostly datacenter proxies though.

Ranking proxies based on real-time performance metrics could make it a game-changer. A web app would definitely make it more accessible—are you planning to add any filtering or automation features for users?

Yes, I am planning on implementing following filtering:
- protocol (http, socks4, socks5)
- location through geolocation databases (MaxMind, IPInfo)
- anonymity (is IP shown in header)
- latency
- trustworthiness (abuseipdb, Spamhaus)
- maybe Bandwidth (evaluated through IPerf3 - needs to be evaluated if it's feasible)
All this metrics added together could then result in a "Proxy quality score" of some type. Am I missing something important?

I will probably add an API at some point, not a priority rn though.

Your response sounds like AI generated blurb. Real time performance? Which metrics will you be using to measure it? No, just ping is not an adequate measure. And how are you even implementing this on a proxy used by possibly hundreds of users? This would be resource intensive task and not worth it for public proxy that will disappear in next hour or so.

Well, how else would you measure a proxies quality? I am currently measuring latency based on response time by timing it (not ICMP pings - could be a valid metric, though)

 

[Panther28]

Do you remember how much money he made/ does the listing or posts of him still exist?

I remember the prices, it was $350 for the software. and $30-50 a month for public always available public proxy service, if you couldn't afford the software.

https://www.blackhatworld.com/search/21204828/?c[users]=NIXMY&o=date

This is the dude I think.

 

[ProxyPapa]

Well, how else would you measure a proxies quality? I am currently measuring latency based on response time by timing it (not ICMP pings - could be a valid metric, though)

Network performance can be examined based on various metrics. Depending on the network's use case and issues, some metrics will be more important to analyse than others. Every network's two main performance measures are throughput and delay. I would also consider bandwidth, speed and throughput. Whenever it is worth for the public proxy measurements, I am not sure. Latency is also a metric here, not the ICMP packets. ICMP packets are sent to measure latency

 

[ristoriel]

Yes, I am planning on implementing following filtering:
- protocol (http, socks4, socks5)
- location through geolocation databases (MaxMind, IPInfo)
- anonymity (is IP shown in header)
- latency
- trustworthiness (abuseipdb, Spamhaus)
- maybe Bandwidth (evaluated through IPerf3 - needs to be evaluated if it's feasible)
All this metrics added together could then result in a "Proxy quality score" of some type. Am I missing something important?

I will probably add an API at some point, not a priority rn though.

That sounds like a great system! Uptime tracking or historical reliability might also be helpful for users looking for consistent performance.

 

[r1ch11r1ch]

That sounds like a great system! Uptime tracking or historical reliability might also be helpful for users looking for consistent performance.

I will implement this with a database in which proxies with their attributes will be inserted regularly to keep a history, which will further evaluate the quality of proxies

 

[ProxyPapa]

I really like this solution from UX UI perspective, Nice and clean with filtering options.

 

[ristoriel]

I will implement this with a database in which proxies with their attributes will be inserted regularly to keep a history, which will further evaluate the quality of proxies

That' a solid one! Having a structured database for tracking performance over time will help in making more informed decisions.

 

[proxygo]

I am in the process of writing a proxy scraper that scrapes proxies from various sources (GitHub, webpages, APIs, google Dorking) and validates them.
Currently, I am getting 200k unique proxies.

The goal is to validate and rank them based on latency, bandwidth, location, privacy and other metrics, so a user can get the fastest nearby proxies for free.
It should be something like ProxyBroker2, but better
I am also planning on adding a webapp, as the tool runs only from the CLI right now.

Happy to hear some thoughts and insights!

proxy scrapers have been around for years from gsa / scrapebox / charon
proxyfire.