PHP CURL with proxies being banned

Hi folks,

I'm developing a free SEO tool that checks PR for thousands of domains with PHP Curl. The problem is that after a while I get banned.

The proxies I'm ussing (http google enabled ones) work great with Scrapebox: I check over 90000 domains with over 50 proxies in 3 hours. But I don't know how Scrapebox server hide this public proxies.

Any experts in this area of knowledge?

Yes, it is normal to get banned as probably you are overusing the proxies in short time and google bans the IP

You need to make a sleep between multiple checks from same proxies and come up with safe algo to rotate proxies for the checks

innozemec said:

Yes, it is normal to get banned as probably you are overusing the proxies in short time and google bans the IP

You need to make a sleep between multiple checks from same proxies and come up with safe algo to rotate proxies for the checks

Click to expand...

and use some elite proxies. public don't hide you...

I hope you are using user agents . coz some hosts ban ips if they don't see comman user agents .

Rotating user agents is important also. I know there are some higher end subscriptions people can get that let them block certain 'signatures' and 'fingerprints' or whatever. I had a custom User Agent once that was scraping about 30,000 websites a day and one day about 2% of them (random, not related all over the US geographically) started failing... Sounds like a small number but it was like 500 websites that started blocking it. Voila, User Agent change fixed it.

gtownfunk