Buzzika
Supreme Member
- Jul 8, 2009
- 1,446
- 1,776
Hello,
I am sure a lot of people already know this. I did not. I usually don't scrape much, but recently I had to scrape tons of keywords for one of my projects.
I started out with ~30 private proxies and pretty soon all of them were banned on Google. I was running just one thread and 19 second delay. : /
Long story short, I decided to scrape all search engines. I knew I would not get the best results but something is better than nothing. Plus I had a VPS just sitting there doing nothing so I had nothing to loose.
To my surprise, even though my proxies were banned on Google, they have been successfully scraping using the "Google API" engine. Also search engines like startpage etc have very similar results to Google, I think they might even be using their APIs.
I am currently running less than 100 threads.
TL;DR: You can use "Google API" search engine in scrapebox with Google banned proxies too. The results might be a bit limited compared to the web version, but worth the effort methinks. Especially if you are scraping for tons of keywords like me.
I am sure a lot of people already know this. I did not. I usually don't scrape much, but recently I had to scrape tons of keywords for one of my projects.
I started out with ~30 private proxies and pretty soon all of them were banned on Google. I was running just one thread and 19 second delay. : /
Long story short, I decided to scrape all search engines. I knew I would not get the best results but something is better than nothing. Plus I had a VPS just sitting there doing nothing so I had nothing to loose.
To my surprise, even though my proxies were banned on Google, they have been successfully scraping using the "Google API" engine. Also search engines like startpage etc have very similar results to Google, I think they might even be using their APIs.
I am currently running less than 100 threads.
TL;DR: You can use "Google API" search engine in scrapebox with Google banned proxies too. The results might be a bit limited compared to the web version, but worth the effort methinks. Especially if you are scraping for tons of keywords like me.
