Massive proxy scraping and validation (200k proxies)

[r1ch11r1ch]

I really like this solution from UX UI perspective, Nice and clean with filtering options.

This looks great, from where is it?

proxy scrapers have been around for years from gsa / scrapebox / charon
proxyfire.

They are quite good from what I heard, have not tested them though. I only looked at open-source scrapers until now

 

[Nabil Sikder]

Is it worth? confused.....

Because this public proxy go down very fast, like checker passed, while connecting it’s down, though some can live more than days.

And some sites load some site doesn’t. In past i tried to test, but i feel like i wasted all the time.

 

[ProxyPapa]

This looks great, from where is it?

I cant really post a link to the resource as it would be classed as advertising. Google - 'public proxy list' and it will be there

 

[r1ch11r1ch]

Is it worth? confused.....

Because this public proxy go down very fast, like checker passed, while connecting it’s down, though some can live more than days.

And some sites load some site doesn’t. In past i tried to test, but i feel like i wasted all the time.

If you do massive scraping operations, you want to automate your proxy retrieval and rotation process.

I want to first implement the scraping of free proxies.
Then to implement APIs of paid proxy providers is not that hard, this will improve the quality of proxies, but still uses the validation code, so you then can rotate and measure your paid proxies

 

[jamie3000]

I've only skimmed through this thread but I'd probably approach this project differently.

I'd probably just adapt this. It can scan the entire internet in 5 min if you have the right hardware.

https://github.com/robertdavidgraham/masscan

 

[r1ch11r1ch]

I've only skimmed through this thread but I'd probably approach this project differently.

I'd probably just adapt this. It can scan the entire internet in 5 min if you have the right hardware.

https://github.com/robertdavidgraham/masscan

I wrote about masscan earlier. It is a valid approach I want to try sometime in the future, especially to find new free proxies.
It will need some more preparation though, internet scanning is a bit complicated (legal issues, find vps that has no problem with it, ...).