[Free Script] Google SERP Scraper + GPT Summarizer

1. Clone the Repository:
   
   
   Code:

   git clone https://github.com/iamBlogPro/Scrape-Summarize.git
   cd Scrape-Summarize

   
   Or Download the ZIP of the file
   
   https://github.com/iamBlogPro/Scrape-Summarize/archive/refs/heads/main.zip
   
   Extract and Open it in VS Code or an IDE of your choice.
   
   
2. Install Required Packages:
   
   
   Bash:

   pip install -r requirements.txt

   
   
3. Download Chromedriver:
   
   Download Chromedriver from here and place it in the same directory as the script.
   
   Now the latest Chrome is super buggy. I recommend using Chome 110 and the chromedriver associated with it (Let me know if you want download links)
   
   
4. Edit the settings.json:
   
   Edit a settings.json file in the root directory with the following content:
   
   
   JSON:

   {
   "openai_api_key": "your_openai_api_key",
   "output_directory": "Output_Folder_Name",
   "keywords_file": "keywords.csv",
   "proxy_file": "proxylist.txt"
   }

   
   
   Replace "your_openai_api_key" with your actual OpenAI API key
   
   
5. In magic.py - add any domain you don't want scraped, or any extensions you don't want or any words in the URL you don't want
   
   You can find this around line 142
   
   
   Python:

   def get_links_with_beautifulsoup(driver):
       logging.info("Retrieving page source for URL extraction")
       skip_domains = ["reddit.com", "istockphoto.com", "groupon.com", "youtube.com", "tiktok.com", "facebook.com", "twitter.com", "petsmart.com", "linkedin.com"]
       skip_extensions = [".gov", ".in", ".uk"]
       skip_words = ["collections", "login", "signin", "advisor", "gov"]

1. Prepare Input Files:
   • Keywords File (keywords.csv): A CSV file containing keywords to search for. Each keyword should be on a separate line without a header.
   • Proxy File (proxylist.txt): A text file containing proxies in the format iport:usernameassword, one per line.
2. Run the Script:
   
   
   Bash:

   python magic.py

1. Initialization:
   • Loads settings from settings.json.
   • Sets up logging to capture script execution details.
2. Proxy Management:
   • Loads proxies from proxylist.txt.
3. Keyword Processing:
   • Reads keywords from keywords.csv.
   • For each keyword, it performs a Google search using Selenium.
4. Content Extraction:
   • Retrieves the page source and uses BeautifulSoup to extract URLs.
   • Filters URLs based on predefined criteria (e.g., domain, extensions, keywords).
5. Data Extraction and Summarization:
   • Extracts content from each URL using Trafilatura.
   • Cleans the extracted content.
   • Summarizes the content using OpenAI's GPT-3.5 Turbo.
6. Output:
   • Saves the summarized content to JSON files in the specified output directory.
   • Updates the keywords.csv file to remove processed keywords.
7. Logging
   The script generates log files in the logs directory, which can be used to monitor its execution and troubleshoot any issues.

• Proxy Management - Automatically loads and manages a list of proxies to avoid IP bans and ensure continuous scraping.
• Headless Browser - Uses Selenium with headless Chrome for efficient and seamless web scraping.
• Content Extraction - Employs Trafilatura to extract and clean textual content from web pages.
• URL Filtering - Filters out irrelevant URLs based on predefined criteria to ensure only relevant content is processed.
• Summarization - Utilizes OpenAI's GPT-3.5 Turbo to generate concise summaries of extracted content.
• Logging - Provides detailed logging for monitoring script execution and debugging purposes.
• Configuration Management - Stores configurable parameters such as API keys, file paths, and directories in a JSON settings file for easy modification.

UltimateBeast said:

Can this be used with GPT-4 instead of GPT-3.5 Turbo?

Click to expand...

BlogPro said:

Yes of course, just edit the model in magic.py

The reason it's set at 3.5 is because most people will want to do this at scale and 3.5 becomes the most cost effective at this point.

Click to expand...

UltimateBeast said:

Got it! Thanks for such a wonderful share..
Most queries nowadays have reddit, quora, Pinterest, or YT videos ranking due to f'ed up G updates.
However, I'll try and check the results first before speculating.

Click to expand...

BlogPro said:

Refer Point No 6 under "How to install"

You can skip domains such as reddit et al

Click to expand...

4440 said:

Cool, I was using gemini pro when it was free with 10s of APIs till I got banned.

Is there any free llm that can run on colab(or not) which can summarize well without losing details?

Click to expand...

BlogPro said:

I'd have to look for LLMs that can run locally (or at a reasonable compute), are free and can still summarise well.

While building this script, I did experiment with a few models - but even on my local Mac, they performed abysmally. Especially with repeated queries, which is what lead me to settle on 3.5

Click to expand...

4440 said:

Yeah, even Gemini pro failed to summarize well just one or two months ago. Hit me up if you find a good one. I'll also leave here if I find one.

A good, free llm that can summarize without losing details is the cornerstone for most of my tasks.

For now I have to use 3.5 as well, Gemini can't follow prompt nicely.

Click to expand...

BlogPro said:

I am actually going to try something this weekend. Let's hope it works.

Click to expand...

ballabee90 said:

thanks for this, want to try, but get this error?ModuleNotFoundError: No module named 'blinker._saferef', can you help me? please

Click to expand...

pip uninstall blinker

pip install blinker==1.7.0

BlogPro said:

Hey,

The Selenium Wire project was archived in January 2024 - so it's not able to support newer deployments. Especially blinker. The blinker._saferef module is unavailable in the latest blinker packages.

So we'll just rollback to blinker < 1.8.0

In your terminal, run

Code:

pip uninstall blinker

Once done, run

Code:

pip install blinker==1.7.0

And it should work

Click to expand...