help regarding google scraping

yellowcat said:

Python:

import requests
import csv
from bs4 import BeautifulSoup

# Read the keywords from a file
with open("keywords.txt", "r") as file:
    keywords = file.read().splitlines()

# Define the User-Agent header
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"}

# Create a new CSV file and write the headers
with open("results.csv", "w", newline="") as file:
    writer = csv.writer(file)
    writer.writerow(["Keyword", "Total Results"])

    # Perform the search for each keyword and write the total number of results to the CSV file
    for keyword in keywords:
        response = requests.get(f"https://www.google.com/search?q={keyword}", headers=headers)
        soup = BeautifulSoup(response.content, "html.parser")
        result_stats = soup.find("div", {"id": "result-stats"})
        if result_stats:
            total_results = result_stats.get_text().split()[1].replace(",", "")
            writer.writerow([keyword, total_results])
            print(f"Keyword: {keyword}, Total Results: {total_results}")
        else:
            writer.writerow([keyword, "Not found"])
            print(f"Keyword: {keyword}, Total Results: Not found")

Code:

keywords.txt (input file)
football
beauty
cinema
cats


results.csv (output file)
Keyword,Total Results
football,3000000000
beauty,6200000000
cinema,2660000000
cats,5720000000

Click to expand...

alifihri said:

Thanks, is there any ways to integrate this with excel or google sheets?

Click to expand...

SirLouen said:

It's a CSV what that script provides

https://developers.google.com/apps-script/samples/automations/import-csv-sheets?hl=en

Click to expand...

alifihri said:

Thanks, I got it and it works, but after searching for 50 keywords almost it shows NOT FOUND in the results, is it limited?

Click to expand...

Steptoe said:

If you're performing all 50 of those quests from 1 IP, Google has probably noticed that you're a bot and blocked you (BeautifulSoup does not execute javascript without being paired with something like Selenium, so it's quite easy to identify).

Click to expand...