Thanks for the answer,
I switched to 1 thread and checked it,
some proxies get blocked in the process, but the ones that do scrape, scrape only 100 per proxy and some of them scrape only 1 url for some reason.
I see here people that scrape thousands of urls and I just can't understand how they do it...
with 70 public proxies (it was 300 in the beggining) I scraped only 700 urls with 6 keywords.
And when I checked the keywords manually, each one had over 500 each..
Hope you could help..
Thanks!
Well it goes in rotation on purpose. So it only uses 1 connection per keyword and it only uses 1 proxy per each request. That way its not hammering on the same proxy over and over again.
The reason it might only get 1 result is that it might only show 1 on that page, if its the last page for a set of results or whatever it can happen.
Also, just because when you search google it says X hundred results, doesn't mean it will show that. Like, you can search for a term and it can say 536 results. Then assuming you have it set to view 100 results per page, for simplicity, and you go to view page 6, which would be the last 36 results, it will kick you back to page 2 and only show you up to say 113 results and say the rest are supplemental. Scrapebox will not scrape supplemental results. This is Google trying to look good on the front side with the numbers it presents, but not really having meaningful results to give you.
If you want my personal opinion, ditch the public proxies all together. Go get 10 private proxies, set your connections to 2 and you can scrape all day long. Its fast because its private proxies, there is no need to monkey with testing, etc... At 2 connections out of 10 proxies they rarely get blocked. Thats only using 20% of your proxies at once. Using this method I have used 38 proxies to scrape for 10 days solid straight, netting over 180 million results. I have done this time and time again.
The point here is private proxies would save you the headache of dealing with public proxies and you would know that your not seeing IP banned or forbidden because the proxy is dead or any number of other issues.
Just my 2 cents.
Also make sure you take into account my video I listed a couple of posts up. If you are using advanced operators in search then thats a completely different ball game.
