Scraping google

emacs2 · Aug 21, 2013

I now tried a lot of stuff and i am not sure if its my fault or its just over these days. I am scraping google with Scrapebox. So i got 10 semi dedicated proxies, used single harvester with random delay and my proxies get banned after say around 30 minutes.

I am just searching for a way to scrape google solid. I dont need millions of urls a day i just scrape for simple stuff like "my house blue". I dont use advanced search operators.

I am a long term SB user i bought it around 2 years ago. Do you guys have any advice or tips for me?

Thank you

innozemec · Aug 21, 2013

get completely dedicated proxies. With the semi-dedicated you might be playing it safe, but you don't know what the others are doing and what crazy settings they are using

emacs2 · Aug 24, 2013

still get banned even with a random delay 50-60 seconds. if dont use proxies and add a delay of 60 seconds i get banned fast too. i am so frustrated.
with what settings do you scrape these days?

scrapefox · Aug 25, 2013

emacs2 said:
still get banned even with a random delay 50-60 seconds. if dont use proxies and add a delay of 60 seconds i get banned fast too. i am so frustrated.
with what settings do you scrape these days?

As innozemec said, just because you're being "gentle" with the proxies, it doesn't mean that someone else isn't abusing them (they're shared after-all).

I generally use private proxies for scraping with a ratio of around 1/10, so if I'm running with 20 proxies for example, I'll use 2 connections.

emacs2 · Aug 25, 2013

hmm... i used 1 connection (1/10) with 10 proxies. i bought another service that said they are not shared.

scrapefox · Aug 25, 2013

emacs2 said:
hmm... i used 1 connection (1/10) with 10 proxies. i bought another service that said they are not shared.

And its working or are you.still running in to problems?

If.it was a new source they were all definitenly google passed before you started?

Posted via Topify using Android

andishm · Aug 25, 2013

emacs2 said:
still get banned even with a random delay 50-60 seconds. if dont use proxies and add a delay of 60 seconds i get banned fast too. i am so frustrated.
with what settings do you scrape these days?

Having semi-dedicated or dedicated proxies for scrapbox i never think its a good idea to invest hundreds of dollars and have them banned and probably your proxy provider is not going to refresh your ip list until next month too

So better use any proxy provider who timely scrape and provide fresh list of proxies which are anonymous and Google passed can be used properly

akacash · Aug 25, 2013

Hello, I've been working on something on the side to dramatically help with this problem. emacs2 if you'd like send me a pm and I'll add you on Skype, I'd like you to test it for me if you'd like. I'd like you to test some proxies, I already know they work and they should work much better too, but I'd like to see just how much and you're perfect to test for me if you'd like. If not no biggie either and good luck. Have a nice day.

emacs2 · Aug 26, 2013

And its working or are you.still running in to problems?

Not working

If.it was a new source they were all definitenly google passed before you started?

Yes all passed

akacash. what exactly are you talking about "something"?
andishm, you got some recommendations?

Scraping google

emacs2

Newbie

innozemec

Elite Member

emacs2

Newbie

scrapefox

Power Member

emacs2

Newbie

scrapefox

Power Member

andishm

Regular Member

akacash

Senior Member

emacs2

Newbie

Main Menu

Marketplace

Making Money

BlackHatWorld

Other