Scrapebox slowing down during harvesting URLs

Hello. I have a problem with my SB. When harvest different URLs, it behaves strangely.
My configuration: i use 50 private proxies, 1000 results per keyword, i set up 1 thread per harvest and 15 sec delay, so my proxies are not banned by Google.
At the very start of harvesting SB usually returns 4-5 pages of URLs with aprox 100 results per page. But after a while(30-60 min) it starts to return less and less results, and the harvesting process during this slows sooooo badly. And after 2-3 hours of harvesting SB returns only 1 page with less then 100 results, and time spended for this 1 page is more then returning 4-5 pages at start.
Proxies are working fine and fast, i check them all the time.
I try to harvest another SE (bing) in parralel with google, and it works fine.
You can see at pictures what i described: example1 is at start, example2 is 2 hours later
What could be the problem?

Your proxies are getting blocked and the more they get blocked the slower it goes as its cycling thru more and more proxies.

The test is only a very basic test, and your using a more complex search string. You can build a custom test for your search string, but thats the issue, your going too fast.

Don't touch your proxies for 48 hours and then try a 60 second delay. If that works dial it back little by little until you start getting blocked again and then you know your sweet spot.

Here is a copy and paste from support that I saved off:

'
There are different kinds of IP bans with Google. The proxy tester only tests against basic keyword search for google. However google can simultaneously allow basic keyword scraping and block other queries, especially advanced operators.



Google can block on many levels for multiple different things. Here is a video by loopline that further explains it:






Additionally this video on how to safely scrape google in 2019 is helpful:






Further you can build a custom test for the exact queries you are using. This will allow you to test your proxies against google and get an accurate result for your query type. Video:


"

Thanks for answer. How many private proxies do you recommend for comfort harvesting? Or how many proxies do i need per 1 thread and with no delay ?

For Google scraping I recommend Storm Proxies since they change IP every request

For Google scraping I recommend Storm Proxies since they change IP every request

simamura said:

Thanks for answer. How many private proxies do you recommend for comfort harvesting? Or how many proxies do i need per 1 thread and with no delay ?

Click to expand...

Id "guess" you would need 75 to 100+ proxies for 1 connection with no delay, but it entirely depends on the actual ip ranges and their history with google plus the queries your using. Its a bit of a moving target that requires some experimentation.

Additionally this video on how to safely scrape google in 2019, should be helpful:



Also as noted above by @710fla you can use storm proxies which is what I use as well.

Thanks all for answers.