Scraping Google
- Thread starter iByte
- Start date
Scritty
Elite Member
- May 1, 2010
- 3,019
- 4,707
A few providers here. Pro proxy, Squid and others. All will start you out with proxies able to scrape Google and provide proper ID cloaking,
However, Google is fecking touchy these days. More than about 20 requests for info in as many minutes, and they will slap a temporary ban on that IP.
My wife has even had a 90 minute ban on our home internet IP just for looking for some stuff to buy for my eldests birthday.
It used to be you'd get a few hundred scrapes before G got twitchy - now it's a few dozen.
If the same job can be done in Yahoo (and most jobs can) I would use them instead.
I'm not saying DON'T scrape G, just be fecking careful these days. Scrapebox on 50+ threads can burn out 25 full private proxies in 5 minutes.
I do scrapes in scrapebox now with i thread per 5 proxies and go into settings and set it to rotate every request - it STILL gets funny if I have a long list of keywords.
12 months ago it was all so darn easy.
Article sites took articles.
Pron forums actually had pron on them
G let you scrape a lot more
Twitter let you hammer automated tools a lot more
CL let you post a lot more
The upside is - most of the lazy idiots are getting out of IM - and there is roughly the same money to round that their always was - so more for those prepared to work.
Waffling now.
/Scritty Out
However, Google is fecking touchy these days. More than about 20 requests for info in as many minutes, and they will slap a temporary ban on that IP.
My wife has even had a 90 minute ban on our home internet IP just for looking for some stuff to buy for my eldests birthday.
It used to be you'd get a few hundred scrapes before G got twitchy - now it's a few dozen.
If the same job can be done in Yahoo (and most jobs can) I would use them instead.
I'm not saying DON'T scrape G, just be fecking careful these days. Scrapebox on 50+ threads can burn out 25 full private proxies in 5 minutes.
I do scrapes in scrapebox now with i thread per 5 proxies and go into settings and set it to rotate every request - it STILL gets funny if I have a long list of keywords.
12 months ago it was all so darn easy.
Article sites took articles.
Pron forums actually had pron on them
G let you scrape a lot more
Twitter let you hammer automated tools a lot more
CL let you post a lot more
The upside is - most of the lazy idiots are getting out of IM - and there is roughly the same money to round that their always was - so more for those prepared to work.
Waffling now.
/Scritty Out
Thanks for the dope, Scritty.
I had used squidproxy and have had these issues and therefore was looking to improve my strategy. I can expand to a large number of proxies but they should be reliable. I get the captcha screen for a random 20 odd IPs from my set of 100. I could replace them, but the result is the same after every attempt.
I don't know if changing proxy providers would solve this.
I had used squidproxy and have had these issues and therefore was looking to improve my strategy. I can expand to a large number of proxies but they should be reliable. I get the captcha screen for a random 20 odd IPs from my set of 100. I could replace them, but the result is the same after every attempt.
I don't know if changing proxy providers would solve this.
What method are you using to scrape? WebBrowser automation or some sockets?
I will be scraping.soon from the UK using WB with rotating cookie profiles and will let you know how it goes.
I would also like to do tests with XForwarded header but i havent built that yet
omida86
Power Member
- Feb 15, 2011
- 791
- 182
Where do you set it to rotate every request when scraping? I can only see that I can rotate after every post...
kokoloko75
Elite Member
- Jan 1, 2011
- 1,627
- 1,950
Scrape only with public proxies.
Use private proxies/VPN/... only for blog commenting.
Beny
Use private proxies/VPN/... only for blog commenting.
Beny
alaltaierii
Supreme Member
- Jun 11, 2010
- 1,494
- 388
You can also scrape with private proxies if you have at least 20-30 and you use a low number of connections.
SkyIsTheLimit
BANNED
- Feb 11, 2012
- 69
- 12
I use public proxies for scraping...
I seem to get captchas with public proxies... I thought these were temporary. Is it possible that a proxy could be permanently black listed?