How do you guys rotate through several proxies to avoid a big website, say Google, from IP / captcha blocking you?
Deathbycapcha has an api. I plan to write some scrapers using javascript (request + cheerio libraries). So using the Google example, I'd guess one could run something like:
TRY assert H1 = "Search Results"
CATCH if false and H2 contains "Please complete this"
EXECUTE deathByCaptcha.getCaptcha().then(postCaptcha())...
So that's captcha handling, but still doesn't answer proxy rotation which I think is an even stronger layer of protection for a scraper.
I'm thinking the try / catch statement would test for an active connection to a whitelisted site. A test fail would enter the catch block and execute an if condition relating to either a bad connection, captcha request, or captcha fail.
I would need a source for proxies though, something either scrapable (free) or very low cost per proxy.
Deathbycapcha has an api. I plan to write some scrapers using javascript (request + cheerio libraries). So using the Google example, I'd guess one could run something like:
TRY assert H1 = "Search Results"
CATCH if false and H2 contains "Please complete this"
EXECUTE deathByCaptcha.getCaptcha().then(postCaptcha())...
So that's captcha handling, but still doesn't answer proxy rotation which I think is an even stronger layer of protection for a scraper.
I'm thinking the try / catch statement would test for an active connection to a whitelisted site. A test fail would enter the catch block and execute an if condition relating to either a bad connection, captcha request, or captcha fail.
I would need a source for proxies though, something either scrapable (free) or very low cost per proxy.
Last edited:
