Hello everyone!
I have a question concerning Google SERP-scraping.
I have multiple class-D networks. During the last 3 months from now, I was able to get the SERPs without being bocked by Google a single time, as the scraping was very slow.
A few days ago, all class-D networks were blocked simultaneously within some minutes and I got a site instead, where you have to enter a captcha. Even after waiting and doing nothing for 24 hours, I now can only fetch one single page per IP without being blocked. When I try to scrape the second site, I am asked to enter a captcha again.
1. Do you have any clue why this happened or suggestions how to solve this problem?
2. What is your waiting-time per IP between fetching two Google pages in order to not get blocked?
3. Do you think it helps to use https to connect to Google?
Thank you very much in advance!
I have a question concerning Google SERP-scraping.
I have multiple class-D networks. During the last 3 months from now, I was able to get the SERPs without being bocked by Google a single time, as the scraping was very slow.
A few days ago, all class-D networks were blocked simultaneously within some minutes and I got a site instead, where you have to enter a captcha. Even after waiting and doing nothing for 24 hours, I now can only fetch one single page per IP without being blocked. When I try to scrape the second site, I am asked to enter a captcha again.
1. Do you have any clue why this happened or suggestions how to solve this problem?
2. What is your waiting-time per IP between fetching two Google pages in order to not get blocked?
3. Do you think it helps to use https to connect to Google?
Thank you very much in advance!
Last edited: