DANG! Google just banned my private proxy!
- Thread starter IMpossible
- Start date
The major factor in searching is the speed in which certain searches come from a specific IP address. So if Google sees 20 searches a second with inurl: per second you are more likely to get hit with a captcha and eventually blocked than if it sees only 1 ever few seconds.
The main issue is to be patient and use single threaded only. It takes time but you can scrape plenty enough.
Another factor in scraping Google, IME, is to use IP addresses for the various datacenters. There are plenty of such lists about but very few of them are up to date and accurate. Get such a list yourself by resolving every google.tld to its IP address and then query that IP address. I'm guessing you get banned from a datacenter but the ban takes some time to propagate to other datacenters which you can in the mean time continue to hit.
The main issue is to be patient and use single threaded only. It takes time but you can scrape plenty enough.
Another factor in scraping Google, IME, is to use IP addresses for the various datacenters. There are plenty of such lists about but very few of them are up to date and accurate. Get such a list yourself by resolving every google.tld to its IP address and then query that IP address. I'm guessing you get banned from a datacenter but the ban takes some time to propagate to other datacenters which you can in the mean time continue to hit.
ija61
Senior Member
- Mar 2, 2011
- 976
- 644
When I saw your post I supposed that "a private proxy" is just a way to say but if you mean that you have only one Proxy then be sure that you will never solve this.
As you can see in loopline video the safe ration is 1 to 10. If you use 1 thread you must have 10 proxy, if you use 2 thread you must have 20 proxy.
But as I said this will not help you very much if you are using search attribute on G since the ban will come faster.
And no, I don't know any way to check if someone else use your proxy but if you are using 1 proxy then be sure that this happen because of you. Scraping with one proxy is like you scrape without proxy.
The difference between a public/shared/private proxy is:
A public proxy are not very reliable (low speed, low up time , high usage rate since they are free)
A shared proxy is a proxy that you share with others clients of your proxy provider (much more reliable than a public one, higher speed, It depends how the other users use those proxy )
A private proxy is a proxy that only you use (high speed, 100% up time, more expensive, and you can control how you use them and how much you abuse them).
My opinion is to use only private proxy and at least 10 of them.
IMpossible
Supreme Member
- Apr 15, 2012
- 1,344
- 304
My intention of getting just one private proxy was to be able to post comments without using my own IP and for scraping without getting a 302 IP block for my own IP.
ija61
Senior Member
- Mar 2, 2011
- 976
- 644
It is ok to use proxy to hide your IP but if you use only one you will only hide your IP but you will not avoid your IP to be banned.
When you are using fast posting or G harvester you send requests to servers (hosting servers / G servers, etc) if you use your own IP or only one proxy then all the request will come from the same IP and any administrator will block any IP that make like 20-30 requests per second since this is way from natural usage.
This is why we use like 10-20 proxy so when we make 20 request they will come from 20 different IP's and this will look like there are 20 users.
Hope this make you a better idea of what proxy are used for. Of course a little research on that will not hurt
If you want a cheap solution, then signup with any VPN service that has an autochange IP address service as part of their client. Then just set that to change every 1 minute or so and then you can happily scrape multi-threaded. Quite possibly cheaper than a private proxy.
You will need to do more work if you want to put this on a server - you'll have to use dynamic DNS service.
You will need to do more work if you want to put this on a server - you'll have to use dynamic DNS service.
IMpossible
Supreme Member
- Apr 15, 2012
- 1,344
- 304
Spent my time studiying all about proxies and scraping, and got the hang of it now. Thank you and everyone else for supporting me!
IMpossible
Supreme Member
- Apr 15, 2012
- 1,344
- 304
They do have a request limit. I'm not sure if it's hourly though. But I had to code around it when building some of my bots.
@impossible
The same thing happened to me. I know the feels XD
Also, private proxys often give out proxies from previous buyers. (still private) However, they have been used.
I found myself having to request a new set of private proxies because the proxies were already blocked by google lol
LOL thats emberassing, but I guess there's no other way around it.
Laubster
Senior Member
- May 21, 2013
- 1,008
- 383
I didn't read everyone else's posts, but long story short you shouldn't be scraping with private proxies. There's no need, and they'll get banned fast. In particular, PR checking will get proxies banned super fast. That's your first problem. Anyways, I scrape public proxies, test them against Google, then plug them into scrapebox and let it run. In general they say your number of connections should be 20% of your total number of proxies. I've got it to scrape as many as 65 URLs per second using public ones. I will say SB's public proxy harvester is horrible out of the box. I use GSA SER to scrape mine, then import into SB.
IMpossible
Supreme Member
- Apr 15, 2012
- 1,344
- 304
Yeah, but when I try scraping with public proxies using the Wordpress footprint, I get 0 results.
- Oct 20, 2012
- 1,529
- 288
PM sent
this ia a very big waste of time
) vpns services like HMA have all their ips shared between their users, so you really do not know how many 1000 of their clients are using the same ips for scraping google )the success rate is very very low, i'm 100% sure of this ...
Last edited:
iglow
Elite Member
- Feb 20, 2009
- 2,254
- 870
goolge now bans very fast on inurl etc queries, you need many priv proxies for scraping [my test of proxies]
DBMEDIALLC
Regular Member
- Feb 24, 2013
- 201
- 44
I've heard that with private proxies you have to run your connections at 10%. 50 private proxies = 5 connections OR100 private proxies = 10 connections. That's generally a safe rule of thumb.
