How to scrape Google SERP 100 results /page ?

Hello

I would like to scrape a lot of SERP Google that show 100 websites in one page.
Here is a sample of URL I want to scrape : http://google.de/search?q=apple+watch&num=100&gl=DE&ie=utf-8&oe=utf-8&safe=off&pws=0&nfpr=1

1) Do I have to have a German proxy in order to see the Google results like a German ?
2) I can scrape URL like that with only one IP every minute. If I do it every 30 sec, a captcha is shown by Google. I created a macro on my browser. It open a URL like above on my browser, wait 55 secondes, delete the cookies et repeat with an other URL.
Does someone have a tips to check Google SERP 100 Results page in less than 60 sec ?
I have already 200 private proxies, but I need to scrape much more Google SERP 100 Results page. @loopline Do you think you could help me ?

Thank you for your help guys =)

You can use scrapebox (it is the best tool I think) or Hrefer from Xrumer (but it needs special configuration).
Using scrspebox I get a tons of links from google and other SERPs, 6000 per hour it isn't anought.

Hello
Do you scrape pages of google with 100 results ? I have Scrapebox.

Yes you have to use a german proxy to see german results, even more it is city based. So you get different results if your proxy is based in frankfurt or berlin, not much difference but its a difference.
Github is helping you alot if you can program some python. Most stuff is already done there, you just need to search for it or hire a freelancer, shouldnt be expensive. The hardest part is the captcha.
Btw why do you need serp check every minute? I need it only everyday.

In python you can try something like that :


You are limited to 100 requests/24H

Batmantheone said:

In python you can try something like that :

Python:

from googleapiclient.discovery import build
my_api_key = "Your API Key”
my_cse_id = "Your CSE ID"

def google_search(search_term, api_key, cse_id, **kwargs):
    service = build("customsearch", "v1", developerKey=api_key)
    res = service.cse().list(q=search_term, cx=cse_id, **kwargs).execute()
    return res
   
result = google_search("Dillion Harper", my_api_key, my_cse_id)
print(result)

You are limited to 100 requests/24H

Click to expand...

Yeah but IME the API results differ wildly from the public ones. Perhaps due to location.

Hire a freelancer to have it done for you.

I can do some search for you for free using scrapebox, if you need.
I use 4 instances all the time so I can share.

Hrefer along with xEvil for Recaptcha and german proxies would work best to scrape the results.

A free chrome extension you could use is Linkclump to export the first 100 results of the SERPs. Link: https://chrome.google.com/webstore/detail/linkclump/lfpjkncokllnfokkgpkobnkbkmelfefj?hl=en But then captchas are still a problem. ;-)

@Toublick You can check out my thread for this - https://www.blackhatworld.com/seo/get-free-google-scraper-with-blacklist-keywords-feature.1242443/
It's a simple JS code that grabs all the URLs from your page. Since it works on client side, it doesn't require any proxies either!