When I scrape google serps I get banned after any requests although I changed proxy ip. Is there any other ban like user agent or cookie ban?
goog uses heurs to detect likely scrapers, pick a simpler/different footprint and have lots of proxies ready lol
Also, look for a google_com.xml file on your hard disk, it's normally hidden so be the net admin when looking. I don't get this file with SB or Hrefer, but I don't know what app you are using to scrape with - it may apply in your case. Delete the file, change your proxy and make sure it's fully annony.
Is there a simply method to clear cookies in java? In which directory is the xml file on a windows vista system when I use Java http requests?
I scrape with my own java software so I cannot turn java off. I cannot override the proxies, are there any java classes available to clear all of them?
you might want to try a VPN. I use mine for scraping and haven't had a problem yet. Make sure its a good VPN though. I may be just lucky, who knows.
Virtual Private Network. I'm not sure if that would work any better than proxies because it basically is like a proxy but is a virtual environment. Its worked for me, but I don't do real heavy scraping. Just google virtual private network.
I use HotSpot Shield for scraping Google and I have to say it does seem to outperform a lot of proxies. At least free proxies. Plus it's so damn easy to change your IP with it.