Google Serp scrape: Ban

[thelastnewbie]

When I scrape google serps I get banned after any requests although I changed proxy ip. Is there any other ban like user agent or cookie ban?

 

[trophaeum]

goog uses heurs to detect likely scrapers, pick a simpler/different footprint and have lots of proxies ready lol

 

[G0D0VERY0U]

Also, look for a google_com.xml file on your hard disk, it's normally hidden so be the net admin when looking. I don't get this file with SB or Hrefer, but I don't know what app you are using to scrape with - it may apply in your case. Delete the file, change your proxy and make sure it's fully annony.

 

[nikoman]

Clear cookies every time you get banned.

 

[thelastnewbie]

Is there a simply method to clear cookies in java?
In which directory is the xml file on a windows vista system when I use
Java http requests?

 

[rocket]

Can you turn java off and then scrape? Never tried.

 

[botdevs]

What nikoman said, clear cookies before you change proxies. It's not anything else.

 

[thelastnewbie]

I scrape with my own java software so I cannot turn java off.
I cannot override the proxies, are there any java classes available to clear
all of them?

 

[rocket]

you might want to try a VPN. I use mine for scraping and haven't had a problem yet. Make sure its a good VPN though. I may be just lucky, who knows.

 

[thelastnewbie]

What do you mean with VPN?

I cannot find this xml file.

 

[Yukinari84]

When I scrape google serps I get banned after any requests although I changed proxy ip. Is there any other ban like user agent or cookie ban?

You need to lots of proxies and clean your cookies every time you burn out an IP.

 

[rocket]

What do you mean with VPN?

I cannot find this xml file.

Virtual Private Network. I'm not sure if that would work any better than proxies because it basically is like a proxy but is a virtual environment. Its worked for me, but I don't do real heavy scraping. Just google virtual private network.

 

[Yukinari84]

Virtual Private Network. I'm not sure if that would work any better than proxies because it basically is like a proxy but is a virtual environment. Its worked for me, but I don't do real heavy scraping. Just google virtual private network.

I use HotSpot Shield for scraping Google and I have to say it does seem to outperform a lot of proxies. At least free proxies.

Plus it's so damn easy to change your IP with it.