r1ch11r1ch
Registered Member
- May 5, 2024
- 66
- 32
This looks great, from where is it?
They are quite good from what I heard, have not tested them though. I only looked at open-source scrapers until now
Massive proxy scraping and validation (200k proxies)
- Thread starter r1ch11r1ch
- Start date
Nabil Sikder
Registered Member
- Feb 25, 2020
- 77
- 25
Is it worth? confused.....
Because this public proxy go down very fast, like checker passed, while connecting it’s down, though some can live more than days.
And some sites load some site doesn’t. In past i tried to test, but i feel like i wasted all the time.
Because this public proxy go down very fast, like checker passed, while connecting it’s down, though some can live more than days.
And some sites load some site doesn’t. In past i tried to test, but i feel like i wasted all the time.
ProxyPapa
Regular Member
- Jan 10, 2025
- 201
- 75
I cant really post a link to the resource as it would be classed as advertising. Google - 'public proxy list' and it will be there
r1ch11r1ch
Registered Member
- May 5, 2024
- 66
- 32
If you do massive scraping operations, you want to automate your proxy retrieval and rotation process.
I want to first implement the scraping of free proxies.
Then to implement APIs of paid proxy providers is not that hard, this will improve the quality of proxies, but still uses the validation code, so you then can rotate and measure your paid proxies
- Jun 30, 2014
- 13,751
- 5,225
I've only skimmed through this thread but I'd probably approach this project differently.
I'd probably just adapt this. It can scan the entire internet in 5 min if you have the right hardware.
https://github.com/robertdavidgraham/masscan
I'd probably just adapt this. It can scan the entire internet in 5 min if you have the right hardware.
https://github.com/robertdavidgraham/masscan
r1ch11r1ch
Registered Member
- May 5, 2024
- 66
- 32
I wrote about masscan earlier. It is a valid approach I want to try sometime in the future, especially to find new free proxies.
It will need some more preparation though, internet scanning is a bit complicated (legal issues, find vps that has no problem with it, ...).