scraping is taking forever

firstnamelastname

Regular Member
Joined
Jun 20, 2015
Messages
410
Reaction score
420
Google is banning proxies so quick these days
The proxies that scrapebox scrapes are basically useless, either they are banned or they get banned within less than 5 minutes.
I have not tried private ones but I would imagine I will face the same problem.
I am trying to scrape 40,000 keywords. How can I do that? right now I scrape a few, turn off, harvest proxies again, start scraping again, turn off, harvest proxies again, it's taking forever.
And I am not using special keywords that trigger the filters like site: or inurl:
I am using normal keywords
how do you guys scrape these days?
 
Use one VPN,you can go with HMA like VPN,and that will be perfect for changing proxies which can not ban your ip.
 
I'm afraid there isn't a perfect option.

Why don't you try using the cloud proxies from ScrapeBox? Those work better.
 
Google is banning proxies so quick these days
The proxies that scrapebox scrapes are basically useless, either they are banned or they get banned within less than 5 minutes.
I have not tried private ones but I would imagine I will face the same problem.
I am trying to scrape 40,000 keywords. How can I do that? right now I scrape a few, turn off, harvest proxies again, start scraping again, turn off, harvest proxies again, it's taking forever.
And I am not using special keywords that trigger the filters like site: or inurl:
I am using normal keywords
how do you guys scrape these days?

Do you have to scrape google or can you use yahoo, bing or some other engine? As thats the quickest path of least resistance. I actually scrape from google and just use private proxies. You would be surprised how fast it gets done even at 1 connection. Or even at 1 connection with a delay. I started up a run last night on 1 server with 50 private proxies and a 5 second delay and scraped over 200K results in about 16 hours. I mean if you only need 40K results, grab 10 proxies, set a delay of 10 in the detailed harvester and just minimize it and come back in a couple days. Or find another engine.

I scrape a lot of results daily, but most of them do not come from google, they come from more creative methods.
 
Google bans a lot of proxies nowadys. Even if you have clean Google passed proxies, you need to ratio very less i.e. around 1 thread per 10 proxies. Or you might try the scanned proxies service of some popular seller such as proxygo to get some good Google passed proxies everyday. And as loopline mentioned, there are other search engines too except Google and you just need to be a bit more creative. Also make sure to update the search engines in Scrapebox by downloading automatically from their servers as there were updates released a couple of times in the last few months or so.
 
Get private proxies

Fix your scraper settings
 
I've been using scrapebox all day with no problems.First and foremost public proxies are horrible and I would invest in private proxies. I recommend buyproxies as I use them with great success and scrape for hours. GL
 
Free proxies are free for a reason. Use private proxies and set your settings so they don't look like your a scraper.
 
Back
Top