Best way to rotate proxies

[Delta223]

How do you guys rotate through several proxies to avoid a big website, say Google, from IP / captcha blocking you?

Deathbycapcha has an api. I plan to write some scrapers using javascript (request + cheerio libraries). So using the Google example, I'd guess one could run something like:

TRY assert H1 = "Search Results"
CATCH if false and H2 contains "Please complete this"
EXECUTE deathByCaptcha.getCaptcha().then(postCaptcha())...

So that's captcha handling, but still doesn't answer proxy rotation which I think is an even stronger layer of protection for a scraper.

I'm thinking the try / catch statement would test for an active connection to a whitelisted site. A test fail would enter the catch block and execute an if condition relating to either a bad connection, captcha request, or captcha fail.

I would need a source for proxies though, something either scrapable (free) or very low cost per proxy.

 

[Googleplex]

How do you guys rotate through several proxies to avoid a big website, say Google, from IP / captcha blocking you?

Deathbycapcha has an api. I plan to write some scrapers using javascript (request + cheerio libraries). So using the Google example, I'd guess one could run something like:

TRY assert H1 = "Search Results"
CATCH if false and H2 contains "Please complete this"
EXECUTE deathByCaptcha.getCaptcha().then(postCaptcha())...

So that's captcha handling, but still doesn't answer proxy rotation which I think is an even stronger layer of protection for a scraper.

I'm thinking the try / catch statement would test for an active connection to a whitelisted site. A test fail would enter the catch block and execute an if condition relating to either a bad connection, captcha request, or captcha fail.

I would need a source for proxies though, something either scrapable (free) or very low cost per proxy.

Hey buddy!
There is many approaches to proxy rotation, the quick n easy one is just use proxychains (if you don't know what is proxychains google "How to Evade Detection Using Proxychains", there is a good tutorial there),
basically you load a list of proxy on the configuration file and it will rotate through the list trying to use each proxy on the list.
the downside of this approach is that if you are using scraped low quality proxies you will lose quite some time trying to connect to non working proxies.