Has Google gone hard on proxies?

[mlkykhan]

I am using 20 private proxies in Scrapebox and had my max connections set to 10% of proxies i.e 2 connections. But after harvesting 40k links google bans the proxies. I tried to harvest with 5% max connections i.e 1 connection. It completed a harvest of 140k links.

Is it just me, my proxies or everyone else is facing difficulty with selecting 10% of proxies as Max connections?

 

[SerpEvil]

i usually use 3-5k public proxies for scraping,and i change them every few hours,so i get millions of urls.20 proxies is a very low number for scraping.

 

[American Trends]

I too face proxy ban most of the time. Even it happens frequently.

 

[cipango]

40 proxies at 1 connection would be a better number for Google.

 

[mlkykhan]

i usually use 3-5k public proxies for scraping,and i change them every few hours,so i get millions of urls.20 proxies is a very low number for scraping.

SB skips a public proxy if it dies during scraping or it doesn't return results for the keyword on which the proxy dies?

I am asking because I heard a lot that scraping with public proxies is a bad idea.

 

[lover]

i usually use 3-5k public proxies for scraping,and i change them every few hours,so i get millions of urls.20 proxies is a very low number for scraping.

What proxy service do you use?

 

[DannyZhang]

Don't use scrapebox if you are doing heavy scrapping. It's great for the other functions but it's not really good when you need to scrape urls in the millions.

You are better off with Gscraper of Hrefer

 

[watsonovedades]

Maybe someting wrong with your proxies?
are they private or semi-private? if they are semi-private you can be sharing the same proxies with heavy spammers marketers and they banned the proxies for all the rest of you (semi.private are shared between 2 or more users)

 

[delta_squadron]

With so few proxies you should avoid scraping with advanced search operators like inurl, etc in order to avoid proxy bans.

 

[BreaknBrix]

It's not rocket science man. You need wayyy more proxies.

I usually scrape around 1000 then replace them every 4 hours or so.

 

[SerpEvil]

What proxy service do you use?

i use Shani service red-proxy.net/

 

[gu3sswh0]

For scraping you are better off using public proxy list that are Google passed.

 

[lover]

i use Shani service red-proxy.net/

Great, i use same service but i scrape with Gscraper.

 

[SerpEvil]

Great, i use same service but i scrape with Gscraper.

gscraper is the only tool i use for scraping and is the best one on the market

 

[mlkykhan]

Maybe someting wrong with your proxies?
are they private or semi-private? if they are semi-private you can be sharing the same proxies with heavy spammers marketers and they banned the proxies for all the rest of you (semi.private are shared between 2 or more users)

Proxies are private. From NewIpNow.org. May be I should contact support.

 

[mlkykhan]

For scraping you are better off using public proxy list that are Google passed.

Which public proxy service do you recommend? I tried scraping lots but only a few were google passed and died very quickly.

 

[laur.laurix]

Dont waste ur money on public proxies they burn fast. Buy 50 private proxies set sb to 2 connections rnd delay min 35 sec max 60 sec ( very important) and fire it up. As we speak my sb is harvesting 60-80 urls/ sec with 2 connections. Pros: the harvesting session is stable, u can scrape for days like this without stopping.
Cons: u wont get 15k urls/s this way

 

[mlkykhan]

Dont waste ur money on public proxies they burn fast. Buy 50 private proxies set sb to 2 connections rnd delay min 35 sec max 60 sec ( very important) and fire it up. As we speak my sb is harvesting 60-80 urls/ sec with 2 connections. Pros: the harvesting session is stable, u can scrape for days like this without stopping.
Cons: u wont get 15k urls/s this way

Thank you. That is why I started with buying 20 private proxies. I guess I will buy some more now. I will test mixing private proxies with share-private proxies too and share result

 

[gary2]

proxie bans happen with everyone... i dont think there is a reason to worry...