[Free Script] Google SERP Scraper + GPT Summarizer

[BlogPro]

Hello lovelies,

Been a while since I dropped something. So am here again, sharing something with you all.

You've shown a lot of love to my Keyword Clustering Script, Keyword Similarity Check and

The Problem

How can we make use of the SERPs to enhance our Content? Enter Google Scraper + Summarizer

The Magic of the script

This script will

1. Scrape Google SERPs
2. Extract top 5 results (configurable)
3. Visit those website and extract the content
4. Summarize the content using GPT 3.5 Turbo
5. Save Summaries to a JSON file with the keyword name. (So the Summar for the keyword "Best SEO Forum" will be saved to "Best SEO Forum.json"
6. Removes all done keywords from keywords.csv

How to install

1. Clone the Repository:
   
   

   git clone https://github.com/iamBlogPro/Scrape-Summarize.git
   cd Scrape-Summarize

   
   Or Download the ZIP of the file
   
   https://github.com/iamBlogPro/Scrape-Summarize/archive/refs/heads/main.zip
   
   Extract and Open it in VS Code or an IDE of your choice.
   
   
2. Install Required Packages:
   
   

   pip install -r requirements.txt

   
   
3. Download Chromedriver:
   
   Download Chromedriver from here and place it in the same directory as the script.
   
   Now the latest Chrome is super buggy. I recommend using Chome 110 and the chromedriver associated with it (Let me know if you want download links)
   
   
4. Edit the settings.json:
   
   Edit a settings.json file in the root directory with the following content:
   
   

   {
   "openai_api_key": "your_openai_api_key",
   "output_directory": "Output_Folder_Name",
   "keywords_file": "keywords.csv",
   "proxy_file": "proxylist.txt"
   }

   
   
   Replace "your_openai_api_key" with your actual OpenAI API key
   
   
5. In magic.py - add any domain you don't want scraped, or any extensions you don't want or any words in the URL you don't want
   
   You can find this around line 142
   
   

   def get_links_with_beautifulsoup(driver):
       logging.info("Retrieving page source for URL extraction")
       skip_domains = ["reddit.com", "istockphoto.com", "groupon.com", "youtube.com", "tiktok.com", "facebook.com", "twitter.com", "petsmart.com", "linkedin.com"]
       skip_extensions = [".gov", ".in", ".uk"]
       skip_words = ["collections", "login", "signin", "advisor", "gov"]

Using the Script

1. Prepare Input Files:
   • Keywords File (keywords.csv): A CSV file containing keywords to search for. Each keyword should be on a separate line without a header.
   • Proxy File (proxylist.txt): A text file containing proxies in the format iport:usernameassword, one per line.
2. Run the Script:
   
   

   python magic.py

How the Script works?

1. Initialization:
   • Loads settings from settings.json.
   • Sets up logging to capture script execution details.
2. Proxy Management:
   • Loads proxies from proxylist.txt.
3. Keyword Processing:
   • Reads keywords from keywords.csv.
   • For each keyword, it performs a Google search using Selenium.
4. Content Extraction:
   • Retrieves the page source and uses BeautifulSoup to extract URLs.
   • Filters URLs based on predefined criteria (e.g., domain, extensions, keywords).
5. Data Extraction and Summarization:
   • Extracts content from each URL using Trafilatura.
   • Cleans the extracted content.
   • Summarizes the content using OpenAI's GPT-3.5 Turbo.
6. Output:
   • Saves the summarized content to JSON files in the specified output directory.
   • Updates the keywords.csv file to remove processed keywords.
7. Logging
   The script generates log files in the logs directory, which can be used to monitor its execution and troubleshoot any issues.

Features of the Script

• Proxy Management - Automatically loads and manages a list of proxies to avoid IP bans and ensure continuous scraping.
• Headless Browser - Uses Selenium with headless Chrome for efficient and seamless web scraping.
• Content Extraction - Employs Trafilatura to extract and clean textual content from web pages.
• URL Filtering - Filters out irrelevant URLs based on predefined criteria to ensure only relevant content is processed.
• Summarization - Utilizes OpenAI's GPT-3.5 Turbo to generate concise summaries of extracted content.
• Logging - Provides detailed logging for monitoring script execution and debugging purposes.
• Configuration Management - Stores configurable parameters such as API keys, file paths, and directories in a JSON settings file for easy modification.

Hope some of you can find this useful

//

Mod Edit: Removed content that is not related to the subforum or topic.

 

[Dopious]

Seems very handy, love the share.

 

[UltimateBeast]

Can this be used with GPT-4 instead of GPT-3.5 Turbo?

 

[Starblazer]

Great share. This will save hours of content research time for each keyword.

 

[BlogPro]

Can this be used with GPT-4 instead of GPT-3.5 Turbo?

Yes of course, just edit the model in magic.py

The reason it's set at 3.5 is because most people will want to do this at scale and 3.5 becomes the most cost effective at this point.

 

[UltimateBeast]

Yes of course, just edit the model in magic.py

The reason it's set at 3.5 is because most people will want to do this at scale and 3.5 becomes the most cost effective at this point.

Got it! Thanks for such a wonderful share..
Most queries nowadays have reddit, quora, Pinterest, or YT videos ranking due to f'ed up G updates.
However, I'll try and check the results first before speculating.

 

[BlogPro]

Got it! Thanks for such a wonderful share..
Most queries nowadays have reddit, quora, Pinterest, or YT videos ranking due to f'ed up G updates.
However, I'll try and check the results first before speculating.

Refer Point No 6 under "How to install"

You can skip domains such as reddit et al

 

[UltimateBeast]

Refer Point No 6 under "How to install"

You can skip domains such as reddit et al

Yeah, that's why I'll test it first by applying all these filters. Let me see how it goes.

 

[damnez]

Great share OP, I will definitely give this a try!

 

[4440]

Cool, I was using gemini pro when it was free with 10s of APIs till I got banned.

Is there any free llm that can run on colab(or not) which can summarize well without losing details?

 

[BlogPro]

Cool, I was using gemini pro when it was free with 10s of APIs till I got banned.

Is there any free llm that can run on colab(or not) which can summarize well without losing details?

I'd have to look for LLMs that can run locally (or at a reasonable compute), are free and can still summarise well.

While building this script, I did experiment with a few models - but even on my local Mac, they performed abysmally. Especially with repeated queries, which is what lead me to settle on 3.5

 

[4440]

I'd have to look for LLMs that can run locally (or at a reasonable compute), are free and can still summarise well.

While building this script, I did experiment with a few models - but even on my local Mac, they performed abysmally. Especially with repeated queries, which is what lead me to settle on 3.5

Yeah, even Gemini pro failed to summarize well just one or two months ago. Hit me up if you find a good one. I'll also leave here if I find one.

A good, free llm that can summarize without losing details is the cornerstone for most of my tasks.

For now I have to use 3.5 as well, Gemini can't follow prompt nicely.

 

[V]

Just wow. I was working on something similar but was stuck at scraping the SERPs. Thank you once again for this awesome share.

 

[BlogPro]

Yeah, even Gemini pro failed to summarize well just one or two months ago. Hit me up if you find a good one. I'll also leave here if I find one.

A good, free llm that can summarize without losing details is the cornerstone for most of my tasks.

For now I have to use 3.5 as well, Gemini can't follow prompt nicely.

I am actually going to try something this weekend. Let's hope it works.

 

[TechEinstien]

Awesome share man.

 

[4440]

I am actually going to try something this weekend. Let's hope it works.

Another question off the topic, but, do you know any good free LLMs that are very consistent and reliable for grammar checks, and also simple rewrite/improvment tasks?

I'd like to use one LLM to replace grammarly checks entirely.

Strict grammar checks are not a problem but I failed to find 7/8b llms that can do simple tasks as well, like rewriting, changing tones, adding headings, etc

 

[KeyProxy]

Great guide

 

[ballabee90]

thanks for this, want to try, but get this error?ModuleNotFoundError: No module named 'blinker._saferef', can you help me? please

 

[BlogPro]

thanks for this, want to try, but get this error?ModuleNotFoundError: No module named 'blinker._saferef', can you help me? please

Hey,

The Selenium Wire project was archived in January 2024 - so it's not able to support newer deployments. Especially blinker. The blinker._saferef module is unavailable in the latest blinker packages.

So we'll just rollback to blinker < 1.8.0

In your terminal, run

pip uninstall blinker

Once done, run

pip install blinker==1.7.0

And it should work

 

[ballabee90]

Hey,

The Selenium Wire project was archived in January 2024 - so it's not able to support newer deployments. Especially blinker. The blinker._saferef module is unavailable in the latest blinker packages.

So we'll just rollback to blinker < 1.8.0

In your terminal, run

pip uninstall blinker

Once done, run

pip install blinker==1.7.0

And it should work

thank you brother working great