Python script

O

onedaydayone

Regular Member
Joined
Aug 5, 2019
Messages
271
Reaction score
91
Does anyone know where I can find a well-written script that scrapes chosen competitors or the top 5 in SERP, for example, and extracts the first 200 words of HTML text to display as results?
 
Advertise on BHW
It seems too custom. You can simply create one by yourself using GPT or Claude.

I use a chrome extension called Easy Scraper but I used to scrape only titles, links of competitor's articles. I don't know if it can help you, but anyway, GPT will do the work for you.
 
onedaydayone said:
Does anyone know where I can find a well-written script that scrapes chosen competitors or the top 5 in SERP, for example, and extracts the first 200 words of HTML text to display as results?
Click to expand...
You can create a Python script using libraries like BeautifulSoup for web scraping and requests for fetching HTML. To scrape competitors or top SERP results, integrate the Google Search API or SERP scraping tools to gather URLs. Ensure your script respects robots.txt and avoids overloading servers to stay compliant with web scraping guidelines.
 
By_ReFLeX said:
I don't think AI can do this.
Click to expand...

Here's what it gave me:


Below is a Python script for scraping the first 200 words of HTML text from the top competitors in a SERP. This script uses the requests, BeautifulSoup, and googlesearch-python libraries. You can install the required libraries with pip install requests beautifulsoup4 googlesearch-python.

Serp Scraper

Python: 
import requests
from bs4 import BeautifulSoup
from googlesearch import search

def get_serp_results(query, num_results=5):
    """Fetch top URLs from Google search."""
    return [url for url in search(query, num=num_results, stop=num_results)]

def extract_first_200_words(url):
    """Extract the first 200 words of visible HTML text from a given URL."""
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()

        soup = BeautifulSoup(response.text, 'html.parser')
        # Extract text from HTML and split into words
        text = soup.get_text(separator=' ', strip=True)
        words = text.split()

        # Get the first 200 words
        first_200_words = ' '.join(words[:200])

        return first_200_words
    except requests.exceptions.RequestException as e:
        return f"Error fetching URL: {e}"

def main():
    # User input for search query
    query = input("Enter your search query: ")

    # Fetch SERP results
    print("Fetching SERP results...")
    urls = get_serp_results(query)

    print("\nTop Competitor URLs:")
    for i, url in enumerate(urls, 1):
        print(f"{i}. {url}")

    print("\nExtracting first 200 words from each URL:\n")
    for i, url in enumerate(urls, 1):
        print(f"--- Competitor {i}: {url} ---")
        snippet = extract_first_200_words(url)
        print(snippet)
        print("\n")

if __name__ == "__main__":
    main()

Key Features of the Script:​

  1. Search SERP: The script uses the googlesearch module to retrieve the top 5 results for a given query.
  2. Extract HTML Text: The first 200 words of visible text are extracted using BeautifulSoup.
  3. Error Handling: Handles connection errors and invalid URLs gracefully.
  4. User-Friendly: Accepts a search query from the user and outputs competitor snippets in a readable format.

How to Use:​

  1. Save the script to a .py file.
  2. Run the script in a Python environment.
  3. Enter a search query when prompted.
  4. Review the extracted text snippets from the top SERP results.

I didn't try the script, but looks working. Maybe it'll need some adjustments after trying. Ai can do everything, you just need to know how to use it :D
 
Danki said:
Bana şunu kazandırdı:


Aşağıda SERP'deki en iyi rakiplerin HTML metninin ilk 200 kelimesini kazımak için bir Python betiği bulunmaktadır. Bu betik, requests , BeautifulSoup ve googlesearch-python kütüphanelerini kullanır. Gerekli kütüphaneleri pip install requests beautifulsoup4 googlesearch-python ile yükleyebilirsiniz.

Serp Kazıyıcı

[KOD=python]ithalat istekleri
bs4'ten BeautifulSoup'u içe aktar
googlesearch'ten aramayı içe aktar

def get_serp_results(sorgu, sonuç_sayısı=5):
"""Google aramasından en iyi URL'leri getir."""
[arama(sorgu, num=num_results, stop=num_results) içindeki url için url] döndür

def extract_first_200_words(url):
"""Belirli bir URL'den görünen HTML metninin ilk 200 kelimesini ayıkla."""
denemek:
yanıt = istekler.get(url, zaman aşımı=10)
yanıt.durum_için_yükseltin()

çorba = BeautifulSoup(yanıt.metin, 'html.parser')
# HTML'den metni ayıklayın ve kelimelere bölün
metin = çorba.get_text(ayırıcı=' ', şerit=True)
kelimeler = text.split()

# İlk 200 kelimeyi alın
ilk_200_kelime = ' '.join(kelimeler[:200])

ilk_200_kelimeyi_geri_döndür
istekler.istisnalar.istekİstisnası hariç e:
"URL alınırken hata oluştu: {e}" döndür

def ana():
# Arama sorgusu için kullanıcı girişi
sorgu = input("Arama sorgunuzu girin: ")

# SERP sonuçlarını getir
print("SERP sonuçları getiriliyor...")
urls = get_serp_results(sorgu)

print("\nEn İyi Rakip URL'leri:")
i için, url in enumerate(urls, 1):
print(f"{i}.{url}")

print("\nHer URL'den ilk 200 kelime çıkarılıyor:\n")
i için, url in enumerate(urls, 1):
print(f"--- Rakip {i}: {url} ---")
kod parçacığı = extract_first_200_words(url)
yazdır(parça)
yazdır("\n")

eğer __name__ == "__main__":
ana()
[/KOD]

Senaryonun Temel Özellikleri:​

  1. SERP'te Arama : Komut dosyası, belirli bir sorgu için ilk 5 sonucu almak için googlesearch modülünü kullanır.
  2. HTML Metnini Çıkar : Görünür metnin ilk 200 kelimesi BeautifulSoup kullanılarak çıkarılır.
  3. Hata İşleme : Bağlantı hatalarını ve geçersiz URL'leri zarif bir şekilde işler.
  4. Kullanıcı Dostu : Kullanıcıdan bir arama sorgusu kabul eder ve rakip parçacıklarını okunabilir bir biçimde çıktı olarak verir.

Nasıl Kullanılır:​

  1. Komut dosyasını .py dosyasına kaydedin.
  2. Betiği Python ortamında çalıştırın.
  3. İstendiğinde bir arama sorgusu girin.
  4. En iyi SERP sonuçlarından çıkarılan metin parçacıklarını inceleyin.

Senaryoyu denemedim ama işe yarıyor gibi görünüyor. Belki denedikten sonra bazı ayarlamalara ihtiyaç duyacaktır. Yapay zeka her şeyi yapabilir, sadece nasıl kullanılacağını bilmeniz gerekir :D
Click to expand...


Congrats buddy. Awesome.
 
BlogPro said:
I wrote a script that scrapes SERPs, scrapes content of the first 5 results, summarizes them using a GPT model, and has proxy support.

See if you find it useful

https://www.blackhatworld.com/seo/free-script-google-serp-scraper-gpt-summarizer.1602000/
Click to expand...
Awesome, thanks you.
 
Danki said:
Here's what it gave me:


Below is a Python script for scraping the first 200 words of HTML text from the top competitors in a SERP. This script uses the requests, BeautifulSoup, and googlesearch-python libraries. You can install the required libraries with pip install requests beautifulsoup4 googlesearch-python.

Serp Scraper

Python: 
import requests
from bs4 import BeautifulSoup
from googlesearch import search

def get_serp_results(query, num_results=5):
    """Fetch top URLs from Google search."""
    return [url for url in search(query, num=num_results, stop=num_results)]

def extract_first_200_words(url):
    """Extract the first 200 words of visible HTML text from a given URL."""
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()

        soup = BeautifulSoup(response.text, 'html.parser')
        # Extract text from HTML and split into words
        text = soup.get_text(separator=' ', strip=True)
        words = text.split()

        # Get the first 200 words
        first_200_words = ' '.join(words[:200])

        return first_200_words
    except requests.exceptions.RequestException as e:
        return f"Error fetching URL: {e}"

def main():
    # User input for search query
    query = input("Enter your search query: ")

    # Fetch SERP results
    print("Fetching SERP results...")
    urls = get_serp_results(query)

    print("\nTop Competitor URLs:")
    for i, url in enumerate(urls, 1):
        print(f"{i}. {url}")

    print("\nExtracting first 200 words from each URL:\n")
    for i, url in enumerate(urls, 1):
        print(f"--- Competitor {i}: {url} ---")
        snippet = extract_first_200_words(url)
        print(snippet)
        print("\n")

if __name__ == "__main__":
    main()

Key Features of the Script:​

  1. Search SERP: The script uses the googlesearch module to retrieve the top 5 results for a given query.
  2. Extract HTML Text: The first 200 words of visible text are extracted using BeautifulSoup.
  3. Error Handling: Handles connection errors and invalid URLs gracefully.
  4. User-Friendly: Accepts a search query from the user and outputs competitor snippets in a readable format.

How to Use:​

  1. Save the script to a .py file.
  2. Run the script in a Python environment.
  3. Enter a search query when prompted.
  4. Review the extracted text snippets from the top SERP results.

I didn't try the script, but looks working. Maybe it'll need some adjustments after trying. Ai can do everything, you just need to know how to use it :D
Click to expand...
Do you have something what helps to scrape AI overview results? (if any is shown)
 
Funiki said:
Man, for what is this?? Scrape content?
Click to expand...

It basically is a way to enhance content. Either manual or AI. With realtime data.

What it does is -

1. Search for the keyword on Google.
2. Visit each of the first 5 pages in the SERPs and extracts content
3. Sends this extracted content to a GPT model of your choice for some content-aware summarization
4. Returns the summary and stores it as JSON with other data/info.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top