Google Serp scrape: Ban

T

thelastnewbie

Newbie
Joined
Dec 6, 2008
Messages
48
Reaction score
0
When I scrape google serps I get banned after any requests although I changed proxy ip. Is there any other ban like user agent or cookie ban?
 
Advertise on BHW
goog uses heurs to detect likely scrapers, pick a simpler/different footprint and have lots of proxies ready lol
 
Also, look for a google_com.xml file on your hard disk, it's normally hidden so be the net admin when looking. I don't get this file with SB or Hrefer, but I don't know what app you are using to scrape with - it may apply in your case. Delete the file, change your proxy and make sure it's fully annony.
 
Is there a simply method to clear cookies in java?
In which directory is the xml file on a windows vista system when I use
Java http requests?
 
Can you turn java off and then scrape? Never tried.
 
What nikoman said, clear cookies before you change proxies. It's not anything else.
 
I scrape with my own java software so I cannot turn java off.
I cannot override the proxies, are there any java classes available to clear
all of them?
 
you might want to try a VPN. I use mine for scraping and haven't had a problem yet. Make sure its a good VPN though. I may be just lucky, who knows.
 
What do you mean with VPN?

I cannot find this xml file.
 
Last edited:
thelastnewbie said:
When I scrape google serps I get banned after any requests although I changed proxy ip. Is there any other ban like user agent or cookie ban?
Click to expand...

You need to lots of proxies and clean your cookies every time you burn out an IP.
 
thelastnewbie said:
What do you mean with VPN?

I cannot find this xml file.
Click to expand...

Virtual Private Network. I'm not sure if that would work any better than proxies because it basically is like a proxy but is a virtual environment. Its worked for me, but I don't do real heavy scraping. Just google virtual private network.
 
rocket said:
Virtual Private Network. I'm not sure if that would work any better than proxies because it basically is like a proxy but is a virtual environment. Its worked for me, but I don't do real heavy scraping. Just google virtual private network.
Click to expand...

I use HotSpot Shield for scraping Google and I have to say it does seem to outperform a lot of proxies. At least free proxies.

Plus it's so damn easy to change your IP with it.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top