PHP CURL with proxies being banned

[tecto]

Hi folks,

I'm developing a free SEO tool that checks PR for thousands of domains with PHP Curl. The problem is that after a while I get banned.

The proxies I'm ussing (http google enabled ones) work great with Scrapebox: I check over 90000 domains with over 50 proxies in 3 hours. But I don't know how Scrapebox server hide this public proxies.

Any experts in this area of knowledge?

 

[innozemec]

Yes, it is normal to get banned as probably you are overusing the proxies in short time and google bans the IP

You need to make a sleep between multiple checks from same proxies and come up with safe algo to rotate proxies for the checks

 

[netbull2007]

Yes, it is normal to get banned as probably you are overusing the proxies in short time and google bans the IP

You need to make a sleep between multiple checks from same proxies and come up with safe algo to rotate proxies for the checks

and use some elite proxies. public don't hide you...

 

[divok]

I hope you are using user agents . coz some hosts ban ips if they don't see comman user agents .

 

[gtownfunk]

Rotating user agents is important also. I know there are some higher end subscriptions people can get that let them block certain 'signatures' and 'fingerprints' or whatever. I had a custom User Agent once that was scraping about 30,000 websites a day and one day about 2% of them (random, not related all over the US geographically) started failing... Sounds like a small number but it was like 500 websites that started blocking it. Voila, User Agent change fixed it.

gtownfunk