Any good ready-made python Google Scraper opensource?
- Thread starter Neylo
- Start date
- Jun 18, 2012
- 536
- 403
Or, is it more feasible to go the Google API way?
https://developers.google.com/custom-search/v1/overview
"
"
https://developers.google.com/custom-search/v1/overview
"
Pricing
Custom Search JSON API provides 100 search queries per day for free. If you need more, you may sign up for billing in the API Console. Additional requests cost $5 per 1000 queries, up to 10k queries per day."
ChillerNiller
Senior Member
- Nov 18, 2019
- 851
- 395
mindeswx
Regular Member
- Dec 5, 2016
- 362
- 589
I am almost certain that search results using this API are different from the real ones
- Jun 18, 2012
- 536
- 403
That’s cheap! Have you used them before?
Preon
Senior Member
- Apr 6, 2016
- 992
- 1,724
There's a number of python libraries specifically designed for searching Google, for example https://pypi.org/project/googlesearch-python/.
You can also search directly on github and fork an existing project.
ifcmd
Registered Member
- May 28, 2022
- 76
- 42
Would also +1 them. Haven't done any high-volume searches, but in initial testing their api, they worked just fine.
Seems less work than having to deal with eventual captchas (+ proxies) doing it manually.
- Jun 18, 2012
- 536
- 403
Have you had a chance to work with it? Does it give you the different SERPs? Such as, PAAs, image links, featured box, similar searches etc...?
- Jun 18, 2012
- 536
- 403
Yeah, this will work I think
Yes I did. Last week I needed to scrap some youtube channels in a specific niche, so I used Python + their API to fetch google results. This way as @ifcmd said, you won't deal with any proxies / captchas.
Keep in mind that sometimes it won't return results for pages 5 - 6 - 7 ect.. I think that this is a limitation from google itself and not their API. You have to use a lot of keywords if you want to scrap big.
Hope it helps
ROHIT KUMAR GAUTAM
Newbie
- Aug 31, 2018
- 46
- 56
Brother you can use python library beutifulsoup , check demo below
import requests
from bs4 import BeautifulSoup
def scrape_google_serp(query):
url = f"https://www.google.com/search?q={query}"
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
}
response = requests.get(url, headers=headers)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
search_results = []
result_divs = soup.find_all("div", class_="g")
for result in result_divs:
title = result.find("h3").text
link = result.find("a")["href"]
snippet = result.find("div", class_="s").text
search_result = {
"title": title,
"link": link,
"snippet": snippet
}
search_results.append(search_result)
return search_results
# Example usage
query = "scraping with Beautiful Soup"
results = scrape_google_serp(query)
for result in results:
print(result["title"])
print(result["link"])
print(result["snippet"])
print()
import requests
from bs4 import BeautifulSoup
def scrape_google_serp(query):
url = f"https://www.google.com/search?q={query}"
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
}
response = requests.get(url, headers=headers)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
search_results = []
result_divs = soup.find_all("div", class_="g")
for result in result_divs:
title = result.find("h3").text
link = result.find("a")["href"]
snippet = result.find("div", class_="s").text
search_result = {
"title": title,
"link": link,
"snippet": snippet
}
search_results.append(search_result)
return search_results
# Example usage
query = "scraping with Beautiful Soup"
results = scrape_google_serp(query)
for result in results:
print(result["title"])
print(result["link"])
print(result["snippet"])
print()
gmailbuyer
Registered Member
- Sep 17, 2022
- 58
- 25
so expensive
seancoder
Failed to resolve a dispute resolution thread.
- Mar 27, 2021
- 129
- 37
Even if one uses the sleep function with this, are there any other caps one should consider and method to circumvent them if any and possible?
D
Deleted member 15724
Guest
Does it have to be with Python? There's scrapebox, octoparse, etc. that can do this pretty much out of the box.
If you need this to deliver to a client, automate or something specific, then there's tons of services that do this with API. Just do your research and find the cheapest option.
If you need this to deliver to a client, automate or something specific, then there's tons of services that do this with API. Just do your research and find the cheapest option.
Emlicorm
Newbie
- Dec 28, 2022
- 49
- 48
Depends what you want to scrape.
I use a modified version of sundios people-also-ask python script and it works very well.