[TIP] How to not get banned while scraping google (important for SB users)

Here is a little secret i decided to share =)
When scraping google i used public proxies (lots of them), call me cheapskate. And there have been a lot of topics saying that private proxies are best for scraping etc. But lets face it, they cost a lot. Public proxies cost next to nothing.
But here is the problem , they get banned a lot, because people used them for scraping.
So how do you scrape without getting banned? even better how to scrape via semi banned proxies?

The fact is that google never actually bans an IP completely, it bans certain requests from it, eg footprints. How do i know this?
I have my own scraping tools that i sometimes share on this forum,and while writing them i came across a strange thing. My scripts couldnt scrape and got asked to enter captcha, while i was googling normally using browsers. And i sought to find out why this was happening.

I tried a LOT of stuff, headers, cookies, requests to some hidden stuff, etc.
And here are the quickest tips for you guys:

Tip 1 (scrapebox does this automatically, but maybe this will be of use to someone):
replace ' ' with +
by default urls get to encode spaces as %20 and google notices that. If you look at any google url you will see only +. It rewrites your quesry to have + instead of %20. So do the same.

Tip 2
This is real gold.

The problem with getting banned is your FOOTPRINT
When searching using footprints that contain stuff like inurl and "powered by wordpress" i got banned almost instantly. Later even when using a browser such queries would produce a captcha.
The problem with scrapebox is that it gives up on a keyword after certain amount of tries. And if you use footprints usually 30% of keywords will fail because the google will ban you for some time, and during that time scrapebox will waste keywords.

I think google also "bans" footprints. I mean that if it spots some suspicious footprints it may ban the IP.

Instead of powered by and inurl concentrate on other texts on page.
for example when i searched for vbulletin forums i didn't use inurl:'member.php' "powered by vbulletin" but instead:
"Home+page"+"find+all+posts+by+this+user"+"about+me" etc.

If you rewrite your footprints to be less "generic" scrapebox will loose much less keywords.
I tried this and my SB had 0 keyword loss with 5000 keywords over public proxies.

Hope this helps.

Yeah, certain footprints will trigger a captcha, and so can going too fast. Without a proxy and one socket in my own scripts I've been able to do millions of searches in a row and never hit the captcha. Even with 2 sockets and no footprint I've been hit.

If you can find good sources for public proxies, like 1,000 at a time you can max out your connection footprints or no footprints.

True, but still no skipped keywords is better than small number of them skipped =)

And there have been a lot of topics saying that private proxies are best for scraping etc. But lets face it, they cost a lot. Public proxies cost next to nothing.
But here is the problem , they get banned a lot, because people used them for scraping.

Click to expand...

Private proxies are best for posting not for scraping and you are correct about the footprints. I mostly avoid using advanced go0gle operators and common footprints.

Regarding the footprints, it is actually the search operators that will get you banned right quick!

actually "powered by" stuff got me banned pretty quick too.

Yeah you might be right though, I just know for sure that the site: operator gets banned real quick.
Wouldn't seem strange for Google to ban such widely searched strings so you might be right buddy.

Great tips! Will try to do some scraping without the operators, might indeed look a bit more natural

So here comes the newbie question, how do you search for inurl or intitle without using the operator? Or do I just need to put a certain amount of time between searches?

Seems interesting man. will give it a try. thanks